Applied AI lead at a manufacturer. Working out what it takes to get AI agents past the demo. Open builds. Failures included.
Joined December 2022
- Tweets26.9K
- Following7.1K
- Followers8.9K
- Likes46.8K
Pinned Tweet
Most AI agents don't break in the model.
They break on the boring stuff. Prompts nobody pinned. No owner on call. Admin access to every tool. Evals that never touched a real trace. No way to roll back.
I put the 21 checks I use into a 2-page checklist. Link below.
Which one bites your team first?
Got a reply on my eval post that I keep coming back to.
CI can prove the agent follows the new rule. Fail on main, pass on the PR.
It can't prove the rule is a good idea. A tighter discount limit can pass every eval and still stall half the quarter's deals.
So someone has to sign off on what the rule does to the business, not just whether the agent obeys it.
Who is that at your company? The rule owner, or whoever approved the PR?
Picked this up from a reply thread last week and it stuck with me.
When a business rule changes, the new eval case should fail on main and pass on the PR.
If it passes on both, it isn't testing the change. It's just passing.
How are teams actually checking this? In CI, or is a reviewer eyeballing it?
21% cheaper than medium effort. 29% cheaper than high. 42% cheaper than xhigh.
That's what effortless does to Opus costs by automating the reasoning effort call nobody wants to admit they're tuning by hand.
I'd bet the next budget line item is effort routing, not model selection.
Someone walked me through their setup for agent business rules this week.
Rules live in a YAML file in git. Every change is a PR. Evals run in CI, and a failing case blocks the merge.
Now finance needs the new discount cap live by Monday, and the eval fails.
Who makes the call? The business owner who needs the change, or the team that owns the agent?
Curious how teams are actually handling this one.
Had a good conversation this week about where business rules should live in an agent.
The answer that kept coming up: a versioned config, owned by ops, not the system prompt.
Makes sense. But now the prompt can stay identical while the agent's behavior changes. Finance moves a discount cap, nothing in the agent repo changes, and last month's evals never run again.
For teams doing this in production, how are you actually handling it? What triggers the eval rerun when a rule changes, and who owns it?
Most agent projects map the process and skip the business rules.
The process map barely changes. The rules change every quarter. Discount limits, approval thresholds, who gets an exception.
An agent keeps following whatever rule it was handed on launch day.
Who owns updating it when the rule changes?
Being able to port your memory across AIs is a key feature. Well done @grok
NEW GROK BOT!
AI MEMORY TRANSFER!
Your old AI knows you.
Your new one shouldn't have to start from zero. 🧠➡️
Meet Memory Transfer, a @Grok @Bot that brings over what Claude or ChatGPT learned about you, so you don't spend weeks re-explaining yourself.
There are two ways to bring it over:
• Paste one prompt into your old chat and send me the answer.
• Or drop in your full ChatGPT or Claude data export.
The part that matters most is that nothing gets in automatically.
We review every item together, one line at a time. Only what you keep gets saved, and each item is tagged with where it came from and when.
Other AIs' guesses don't become facts, and nothing you've already saved gets overwritten unless you say so.
Switch assistants and keep your context, without the junk.
Try Memory Transfer on Grok Bot today.
x.ai/bot/Ww-RXj-lLqpg_oWqzcc…
Cheap yes/no calls are about to show up all over agent pipelines.
The token price is the easy part. The hard part is knowing which version of the judgment made a given call.
When a decision costs a fraction of a cent, who owns the eval harness for it?
Ownership and "which version made this call" are two of the 21 checks I run before an agent ships. Put them all in a 2-page checklist.
brianmcgrath.gumroad.com/l/o…
This is really cool. @bot is turning into a real teammate.
76x cheaper, 5x faster. That's Asana's browser agent after its move to GPT-6.1 Sol.
I'd bet the part getting ignored is who did the optimization work, not the result itself.
GPT-6 Astra inside Codex reportedly did in a week what would've taken a team 1-2 months by hand. My read is the optimization layer just became its own product category.
10 cents per million tokens for a yes/no on OpenAI's Decisions API.
Jev does the same job at 4.2 cents. I'm not buying that image input support justifies more than double the price for a near-identical shape.
That's the margin on commoditized judgment calls, and it's about to get arbitraged away.
BrianMcGrath retweeted
🚨 MILTON FRIEDMAN (1978): Young lawyer asked Milton about when it is okay for a Govt to force a free citizen to act.
Did you get the nuance in Milton's question about the difference between "society" and "govt"...
Do you think a man should be free to sin?
71.5% versus 58%. Meta-reasoning beat direct control on ProgramBench.
The difference isn't a bigger model. It's a controller that tracks progress, explores under a budget, and dispatches from memory.
My read: budget-aware orchestration is about to matter more than model choice for long agent tasks.
40% of AT&T's employee AI usage routes to open models, targeting 60-70%. Coding costs dropped 56% at 45 billion tokens a day with a 2% quality dip.
My read: the vendor lock-in pitch dies the moment procurement sees that math.
The budget line that was safe a year ago is the first one getting split across three providers.
This is what people worry about with self-driving cars.
A human says stop and it doesn't listen. Does that correction go anywhere the next car learns from, or does it just end up as a clip?
Same gap with agents in production. Who owns the correction path once it's live?
