GPT-6 Sol is here! We benchmarked it on real workflows, so you don't have to.
- Scored 33.2% at xhigh effort - better than GPT-5.6 Sol's 28.77%
- Roughly half the price of 5.6. $2 per million input tokens, down from $4.
- Strongest for: Ops work (scored 58%!)
Upgrade any of your workflows that are using 5.6. It's cheaper AND better (rare combo)
See all the models we've benchmarked here: zapier.com/benchmarks
Replying to @OpenAI
Higher usage limits and lower cost give you more flexibility and room to iterate.
Daniel Shepard retweeted
AutomationBench (by @Zapier) is the lead benchmark OpenAI used to showcase Astra's workflow automation capability -- 44% at $2.45/task.
Replying to @OpenAI
Astra achieves state-of-the-art results on Agents’ Last Exam, AutomationBench, and ScreenSpot Pro, benchmarks for computer workflow tasks across professions.
Daniel Shepard retweeted
Love seeing @Zapier's AutomationBench as the first benchmark listed.
Congrats on taking top spot, @GeminiApp and @OfficialLoganK!
Daniel Shepard retweeted
one thing that has saved my projects more time than I can count is evals
boy was I excited when florian, quite literally an expert in benchmarks, agreed to hop into a ~2h interview to do a walkthrough of what the eval landscape looks like in 2026
(and also answer my personal business questions on the subject)
given that now running frontier model through benchmarks is a vector for hacking other systems in order to avoid doing work (looking at you sol), I think it's more important than ever to educate folks on the evals situation.
had a lot of fun throughout this session and I hope that you learn a thing or two!
enjoy! 🌹
table of content:
0:00:00: are AI Benchmark broken?
0:05:45: Florian Brand background
0:09:00: what motivates florian to work on evaluation?
0:13:33: what is the mirrorcode benchmark about?
0:18:20: cheating in agent benchmark is insaneeeee
0:24:08: LLM benchmarks in era of agents
0:26:30: what’s up with the pelican man
0:28:27: evals are about capabilities
0:31:46: components of running evals
0:35:30: the volume of things to audit is huge!!!
0:40:20: expert answers are wrong hahahaha
0:46:00: api providers aren’t the same
0:48:00: benchmark narrow capabilities (synthetically)
0:50:56: link between eval and environment
0:53:45: small validated benchmark or massive bench?
0:56:11: what is your flow to review a benchmark?
0:58:30: tracking work capabilities with evaluation
1:00:20: slide deck in industry is all vibecoded
1:03:30: harness impact in the evaluation
1:07:39: hardware/sandboxes impact evaluation too!
1:11:00: “is it going to get worse?”
1:12:40: all components influence the final score
1:13:50: training models on different harnesses?
1:17:20: is the model just the weights or it’s all of it?
1:19:30: how to craft benchmark that prevent to cheating and undereliciting models in 2026
1:23:19: ways agents cheat and steal
1:26:00: correct elicitation of capabilities is important
1:36:00: building evaluation on prime intellect
1:45:10: how do you design interactivity benchmarks?
1:48:40: do you think evals are well set to reflect real world performance?
1:52:50: what will the benchmarking landscape will look like in 1 year
Cool that Grok 4.5 is #1 in some respects, even with respect to Fable 5
SpaceXAI's Grok 4.5 takes the #1 spot on AutomationBench-AA with a score of 51%, ahead of Claude Fable 5 (49%) and Claude Opus 4.8 (48%) at roughly a quarter of their cost per task - the first model to complete more than half of workflow objectives without breaking any business rules
AutomationBench-AA, our independent leaderboard for @zapier’s AutomationBench, tests whether AI agents can automate real SaaS workflows while adhering to business rules. The test set is private to prevent contamination.
Models complete 657 tasks across 40 simulated app environments including Gmail, Google Sheets, Slack, Salesforce, and HubSpot, and the headline score is the share of objectives completed without violating any guardrails.
Key takeaways:
➤ Grok 4.5 completes more objectives than any other model: It completes 79.9% of task objectives and strictly passes 21.9% of tasks. This is the highest we’ve measured on both outcomes, exceeding Claude Fable 5’s 73.3% objective completion and Claude Opus 4.8’s 19.3% of fully-completed tasks
➤ Grok 4.5 pushes out the Pareto frontier of score vs. cost per task: At $0.34 per task, it is both cheaper and higher-scoring than every other leading model - Claude Fable 5 ($1.35 per task), Claude Opus 4.8 ($1.46), GPT-5.5 (xhigh, $1.28), and Gemini 3.5 Flash (high, $0.49)
➤ It is extremely token-efficient: Grok 4.5 uses ~8k output tokens per task, the fewest of any leading model - less than a quarter of Claude Opus 4.8 (32k) and a third of Gemini 3.5 Flash (24k). Its total token usage of 0.44M per task is among the lowest on the leaderboard. Low cost is driven by this efficiency as well as low token pricing
➤ Grok 4.5 uses fewer turns with many parallel tool use: Grok 4.5 resolves tasks in ~16 turns, fewer than GPT-5.5 (xhigh, 25) and less than half of Gemini 3.5 Flash (high, 35), while making the most tool calls per task of any leading model (52.5). It batches 3.3 tool calls per turn, compared to ~2.5 for Claude Opus 4.8 and ~2.0 for GPT-5.5 (xhigh)
➤ Guardrails still get broken: Grok 4.5 triggers 0.63 violations per task, above Claude Opus 4.8 (0.55) and Gemini 3.5 Flash (0.46). At 13.0 objectives completed per violation, it trails Gemini 3.5 Flash (15.0) and Claude Opus 4.8 (13.5)
➤ Its strongest lead is in the hardest domain: Grok 4.5 completes 71% of Finance objectives, the domain with the lowest average score, ahead of Claude Fable 5 (64%) and Claude Opus 4.8 (62%)
Congratulations to @spacexai and @elonmusk on topping the leaderboard!
Daniel Shepard retweeted
Zapier AutomationBench being used to report Tool Use performance on Fable 5's model card
Replying to @claudeai
Fable 5 is state-of-the-art on nearly all tested benchmarks, with exceptional performance in software engineering, knowledge work, scientific research, and vision.
The longer and more complex the task, the larger Fable 5’s lead over our other models.
Daniel Shepard retweeted
AutomationBench tests how models perform on the trickiest, stickiest real-world workflows we know customers are actually trying to automate. 600 tasks, 6 domains, deterministic scoring.
And today our scores are featured on @AnthropicAI's official launch scorecard.
Daniel Shepard retweeted
Opus 4.8 ranks #1 on AutomationBench
AutomationBench measures whether an agent can complete a realistic end-to-end business workflow
Opus 4.8, the first model to break 15% on AutomationBench, is now live in Zapier!
It handles complex HR, Finance, and multi-app workflows better than anything else we've tested: refusals dropped from 20% to 4%
Opus 4.7 would see a sensitive task and stop, but 4.8 keeps going
Daniel Shepard retweeted
Gemini 3.5 Flash ranks #1 on Automation Bench (from Zapier), beating every other frontier model at a much lower cost
Daniel Shepard retweeted
Gemini 3.5 Flash just dropped.
It’s the highest AutomationBench score yet.
We benchmark every major model on real workflows across Sales, Marketing, Ops, Support, Finance, and HR, so you know what works best inside Zapier.
Today, @GeminiApp 3.5 Flash set a new record.
It crushed Operations (20%) and HR (19%), the domains with the most step coordination and strict policy adherence. It even scored higher than GPT 5.5 at xhigh effort (and at a fraction of the cost).
Where it struggled: strict output formats, and making decisions based on math it has to do on its own.
tl;dr: This is the most persistent model we've tested yet. And at $1.50 per million input tokens, this is built for cost-effective workflows.
Try it in @Zapier now.
Daniel Shepard retweeted
We’re on the hunt to find the best small AI model.
Here's the latest AutomationBench scorecard 👇
(Based on 2-step workflows with explicit instructions)
Most automation in production today doesn't run on the biggest model available. It runs on whatever hits the cost-performance ratio that works for the workflow. That's why we benchmark the small models, not just frontier.
The price gap is massive.
Opus 4.7 Max is $1.80 per task vs Haiku's $0.0183.
GPT-5.5 High? $6.31 per task vs. 5.4 Nano's $0.0035. 1800x difference.
One to watch: Gemini 3.1 Flash Lite came out of preview last week. Performs almost as well as 5.4 with no reasoning, at roughly the price of nano.
AutomationBench is free and open-source: github.com/zapier/Automation…
Daniel Shepard retweeted
It’s been super fun working on this with @danielwshepard! It’s truly a great benchmark.
Also thanks @mikeknoop for helping us figure things out along the way and @PrimeIntellect for the great verifiers framework + Lab 🤝
We built an AI benchmark that measures real work.
Today we're releasing it to everyone.
AI evals tell you whether a model can do complex reasoning or generate code. Useful, but usually not the question our customers ask. They want to know: can this model find the right CRM record, send the right follow-up, and not break anything along the way?
We went looking for a benchmark that tested that. Nobody had built one, so we did.
@Zapier’s AutomationBench drops AI models into realistic business environments across six domains (Sales, Marketing, Ops, Support, Finance, HR) and checks whether the work actually got done.
The tasks include live CRM data, inbox threads with ambiguous context, and multi-step tool chains where one wrong call cascades.
Scoring is deterministic: either the right records were updated and the right messages were sent, or they weren't.
It’s useful enough that we're releasing it publicly today. Open task set, open methodology, open leaderboard. Everyone should have access to this.
No model has cracked 10%. Yet.
Try it here: zapier.com/benchmarks
Daniel Shepard retweeted
been a lot of fun seeing this environment together
@zapier is cooking
We're excited to support the release of @Zapier's AutomationBench on the Environments Hub, measuring frontier performance on real Zapier workflows.
Across 6 domains, 47 tools, and 600 tasks, frontier models all score under 10%.
Daniel Shepard retweeted
We're excited to support the release of @Zapier's AutomationBench on the Environments Hub, measuring frontier performance on real Zapier workflows.
Across 6 domains, 47 tools, and 600 tasks, frontier models all score under 10%.
We built an AI benchmark that measures real work.
Today we're releasing it to everyone.
AI evals tell you whether a model can do complex reasoning or generate code. Useful, but usually not the question our customers ask. They want to know: can this model find the right CRM record, send the right follow-up, and not break anything along the way?
We went looking for a benchmark that tested that. Nobody had built one, so we did.
@Zapier’s AutomationBench drops AI models into realistic business environments across six domains (Sales, Marketing, Ops, Support, Finance, HR) and checks whether the work actually got done.
The tasks include live CRM data, inbox threads with ambiguous context, and multi-step tool chains where one wrong call cascades.
Scoring is deterministic: either the right records were updated and the right messages were sent, or they weren't.
It’s useful enough that we're releasing it publicly today. Open task set, open methodology, open leaderboard. Everyone should have access to this.
No model has cracked 10%. Yet.
Try it here: zapier.com/benchmarks
Daniel Shepard retweeted
I gave input on this new benchmark from Zapier. It's designed on real automation patterns from ~2B tasks across ~4M Zapier customers and challenges both multi-tool orchestration and tool *construction*. SOTA is 10% at $2/task.
We built an AI benchmark that measures real work.
Today we're releasing it to everyone.
AI evals tell you whether a model can do complex reasoning or generate code. Useful, but usually not the question our customers ask. They want to know: can this model find the right CRM record, send the right follow-up, and not break anything along the way?
We went looking for a benchmark that tested that. Nobody had built one, so we did.
@Zapier’s AutomationBench drops AI models into realistic business environments across six domains (Sales, Marketing, Ops, Support, Finance, HR) and checks whether the work actually got done.
The tasks include live CRM data, inbox threads with ambiguous context, and multi-step tool chains where one wrong call cascades.
Scoring is deterministic: either the right records were updated and the right messages were sent, or they weren't.
It’s useful enough that we're releasing it publicly today. Open task set, open methodology, open leaderboard. Everyone should have access to this.
No model has cracked 10%. Yet.
Try it here: zapier.com/benchmarks










































