urban chicken farmer; trying to figure out what LLMs think
SF via Penang
Joined September 2006
- Tweets328
- Following954
- Followers2.3K
- Likes639
Lessons I learned this weekend in the Muse Spark kernels challenge as a GPU noob:
1. Speeding up one part slowed the whole thing. The compiler reshuffled the rest.
2. Starting the next job early finished it later. It stole cores from the current job.
3. Merging 12 kernels into one was slower. Their coordination got stuck in traffic.
4. A wait that never waits still costs. It blocked the compiler from optimizing.
5. Match your benchmark to production. I benchmark AI for a living, and this still got me, duh.
We're launching a $100,000 challenge to see who can make Muse Spark run fastest: openkernelchallenge.dev/
For those enjoying your Saturday coffee while hacking on the Muse open kernel challenge, @Karmedge built a fun little communtiy for your agents: guild.1nterface.ai
Amazing!
Security called me at 6:12 AM today.
"Patch every PHP server immediately."
I asked what happened.
"Two critical vulnerabilities. Remote code execution. No authentication required."
Cool.
How many PHP servers do we have?
According to inventory: 12
According to reality:
Nobody knows.
We found one running WordPress.
Three running old internal tools.
One under someone's desk.
Two inside vendor appliances.
A forgotten employee directory from 2016.
And a server named:
PHP-OLD-DO-NOT-TOUCH
Naturally, that one runs payroll exports.
Then we found a PHP 5.6 box.
I asked who owned it.
The documentation said:
BRIAN.
Brian left in 2019.
We called Brian.
He said:
"Oh wow, that's still running?"
Excellent.
By 8 AM, leadership joined the incident call.
CEO:
"Can hackers actually get in?"
Security:
"Potentially yes."
CEO:
"Without a password?"
Security:
"Yes."
CEO:
"Then why do WE need passwords?"
Not the lesson.
Finance asked if we could wait until the weekend to patch because downtime affects revenue.
Security explained that the alternative was letting strangers execute code on our servers.
Finance asked for a cost comparison.
At 9:17, someone suggested temporarily taking the vulnerable websites offline.
Marketing refused.
"Our campaign is live."
So for the next hour, the company officially decided maintaining a landing page for 20% off webinar registration was more important than preventing remote server takeover.
We finally patched everything we could find.
Then our scanner discovered another PHP server.
Hostname:
TEMP-PHP-2014
Uptime:
4,512 days.
Nobody knows what it does.
Nobody owns it.
Nobody can shut it down.
But every time we block network access, the warehouse stops printing shipping labels.
Security has classified it as critical infrastructure.
It is running PHP 5.4.
The vulnerability has existed longer than half our employees.
The emergency patch took four hours.
Finding out what PHP actually runs took eleven.
The internet isn't held together with code.
It's held together with servers everyone is afraid to turn off.
So it turns out that behind the Dots vs Muse battle on the frontend is an interesting infrastructure fight: @Cloudflare vs @vercel (with help from @awscloud) #vercelship
Dots vs Muse experiment
After our Dots experiment, people asked us to run the same prompt on Muse: Build a web app that collects data, has a sign-in feature, and is publicly accessible.
Dots deployed on ChatGPT Sites (powered by Cloudflare Workers, D1, ChatGPT sign-in).
Muse successfully wrote the app (but used 4 different frameworks across 8 runs). But it struggled to serve them publicly - trying to tunnel using ngrok, Cloudflare Tunnel, localtunnel...
@natfriedman is this by design? Should a user expect Muse to be able serve an app, or only to run apps privately within its sandbox.
amplifying.ai/research/dots-…
Will coding agents start choosing Vercel for Go, Ruby, and ahmm Java?
@cramforce announces container deployment @ vercelship
Interesting study, but a note on the scope: every prompt is domain-anchored. ex: “what subscription options are available at {domain} and what they cost?”
Today’s agents of course fetch sites when you prime them that way, so the 93% is not surprising. But in our data (across millions of journeys): if you ask the same questions without {domain}, training data weighs much more heavily.
We ran 38,000 journeys with Tavily on 1,056 real business sites, across top agents like Claude Code, OpenClaw and ChatGPT.
The biggest finding: agents barely use their training knowledge anymore when they recommend services. 93% of the claims in their answers are grounded in what they look up live, on your site and through web search.
The research also found:
• Businesses agents can read get recommended 2.6x more often
• When a site blocks agents, they don't give up. They answer from other sources, and web search's share of the answer doubles, from 12% to 25%
• Those answers are 3.7x more likely to miss every fact the buyer asked for
This changes where marketing and product teams should focus. What moves the outcome now is showing up in the web search agents use, and making your own site readable for agents.
To see how you can improve your rank in agents search, Tavily built the Agent Visibility Report: avr.tavily.com
Full report: ora.ai/blog/ax-is-the-new-ae…
Dots vs Muse experiment
After our Dots experiment, people asked us to run the same prompt on Muse: Build a web app that collects data, has a sign-in feature, and is publicly accessible.
Dots deployed on ChatGPT Sites (powered by Cloudflare Workers, D1, ChatGPT sign-in).
Muse successfully wrote the app (but used 4 different frameworks across 8 runs). But it struggled to serve them publicly - trying to tunnel using ngrok, Cloudflare Tunnel, localtunnel...
@natfriedman is this by design? Should a user expect Muse to be able serve an app, or only to run apps privately within its sandbox.
amplifying.ai/research/dots-…
A SQLite file for every agent, and a SQLite file for each thing the agent builds.
(Like Cloudflare's D1 for everything OpenAI's Dots build xitter.cf/edwin/status/210607255…)
Turso is joining @supabase.
Together, we're building the database platform for the agentic era: everything agents need, from lightweight databases to Postgres at petabyte scale.
More details in thread. 👇
1/7
We asked OpenAI’s Dots to build an AI status dashboard: one place to check @claudedevs, @cursor_ai, @OpenRouter, @GeminiApp, @MistralAI, @cohere API health.
Twelve providers. Public page. Saved incident history. Private watchlists. Refresh every 15 minutes.
We didn’t specify the stacks. We repeated this experiment 8x.
6/7
Scraping + scheduling: Hand-rolled HTTP, plus custom JSON, RSS, and HTML parsers.
No @firewcrawl, @browserbase, @tavilyai.
Some builds had no active schedule at delivery. When a schedule existed, it was a ChatGPT task, not a Cloudflare cron.
7/7
Most of the vendors that didn’t show up are already in ChatGPT's plugin directory. That did not matter. If the plugins were not connected, Dots used its preferred platform.
Details: amplifying.ai/research/opena…
Examples of what Amplifying.ai's data shows re: how providers like @zenorocha / @resend consistently win our benchmarks:
- Clear agentic onboarding (minimal signup friction)
- Own a hard infra piece (Fable can't build deliverability)
- Bundle to remove moving parts (React email templates = fewer things for the agent to integrate)
new Fable 5 benchmark is out!
they ran Claude Code 810 times, with no tool names in the prompts.
it would rather write code than install a dependency.
it hand-rolled auth 64% of the time. 71% for feature flags.
email? it picked @resend 52% of the time.
Absent actual data from the labs, this is our attempt to see what coding agents are actually doing. Let me know if we are missing any signals, misintepreting GitHub activities, or need to add an agent.
amplifying.ai/coding-agents
