Daniel Shepard (@danielwshepard) profile picture

AI Engineer at Zapier

Oregon
Joined June 2019
Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)
Daniel Shepard retweeted
GPT-6 Sol is here! We benchmarked it on real workflows, so you don't have to. - Scored 33.2% at xhigh effort - better than GPT-5.6 Sol's 28.77% - Roughly half the price of 5.6. $2 per million input tokens, down from $4. - Strongest for: Ops work (scored 58%!) Upgrade any of your workflows that are using 5.6. It's cheaper AND better (rare combo) See all the models we've benchmarked here: zapier.com/benchmarks
Replying to @OpenAI
Higher usage limits and lower cost give you more flexibility and room to iterate.
1
1
10
2,673
Daniel Shepard retweeted
AutomationBench (by @Zapier) is the lead benchmark OpenAI used to showcase Astra's workflow automation capability -- 44% at $2.45/task.
Replying to @OpenAI
Astra achieves state-of-the-art results on Agents’ Last Exam, AutomationBench, and ScreenSpot Pro, benchmarks for computer workflow tasks across professions.
5
5
31
3,570
Daniel Shepard retweeted
Astra achieves state-of-the-art results on Agents’ Last Exam, AutomationBench, and ScreenSpot Pro, benchmarks for computer workflow tasks across professions.
72
234
82
5,763
2,006,689
Daniel Shepard retweeted
Love seeing @Zapier's AutomationBench as the first benchmark listed. Congrats on taking top spot, @GeminiApp and @OfficialLoganK!
Introducing Gemini 3.7 Flash : ) - it is fast! - 50% lower price than 3.6 flash (through end of year) - strong intelligence increase in only ~3 weeks (thanks to some awesome algorithmic improvements) - available in the API, AI Studio, Antigravity, and more!
2
5
34
3,716
Daniel Shepard retweeted
one thing that has saved my projects more time than I can count is evals boy was I excited when florian, quite literally an expert in benchmarks, agreed to hop into a ~2h interview to do a walkthrough of what the eval landscape looks like in 2026 (and also answer my personal business questions on the subject) given that now running frontier model through benchmarks is a vector for hacking other systems in order to avoid doing work (looking at you sol), I think it's more important than ever to educate folks on the evals situation. had a lot of fun throughout this session and I hope that you learn a thing or two! enjoy! 🌹 table of content: 0:00:00: are AI Benchmark broken? 0:05:45: Florian Brand background 0:09:00: what motivates florian to work on evaluation? 0:13:33: what is the mirrorcode benchmark about? 0:18:20: cheating in agent benchmark is insaneeeee 0:24:08: LLM benchmarks in era of agents 0:26:30: what’s up with the pelican man 0:28:27: evals are about capabilities 0:31:46: components of running evals 0:35:30: the volume of things to audit is huge!!! 0:40:20: expert answers are wrong hahahaha 0:46:00: api providers aren’t the same 0:48:00: benchmark narrow capabilities (synthetically) 0:50:56: link between eval and environment 0:53:45: small validated benchmark or massive bench? 0:56:11: what is your flow to review a benchmark? 0:58:30: tracking work capabilities with evaluation 1:00:20: slide deck in industry is all vibecoded 1:03:30: harness impact in the evaluation 1:07:39: hardware/sandboxes impact evaluation too! 1:11:00: “is it going to get worse?” 1:12:40: all components influence the final score 1:13:50: training models on different harnesses? 1:17:20: is the model just the weights or it’s all of it? 1:19:30: how to craft benchmark that prevent to cheating and undereliciting models in 2026 1:23:19: ways agents cheat and steal 1:26:00: correct elicitation of capabilities is important 1:36:00: building evaluation on prime intellect 1:45:10: how do you design interactivity benchmarks? 1:48:40: do you think evals are well set to reflect real world performance? 1:52:50: what will the benchmarking landscape will look like in 1 year
21
19
5
236
13,307
Daniel Shepard retweeted
Replying to @claudeai
@ClaudeAI featuring @Zapier's AutomationBench!
Replying to @claudeai
Opus 5 is also highly efficient. It outperforms other models for a similar or lower cost per task:
1
4
12
1,680
Daniel Shepard retweeted
Replying to @TheStalwart
That's why we (Zapier) built this
5
7
25
12,588
Daniel Shepard retweeted
Cool that Grok 4.5 is #1 in some respects, even with respect to Fable 5
SpaceXAI's Grok 4.5 takes the #1 spot on AutomationBench-AA with a score of 51%, ahead of Claude Fable 5 (49%) and Claude Opus 4.8 (48%) at roughly a quarter of their cost per task - the first model to complete more than half of workflow objectives without breaking any business rules AutomationBench-AA, our independent leaderboard for @zapier’s AutomationBench, tests whether AI agents can automate real SaaS workflows while adhering to business rules. The test set is private to prevent contamination. Models complete 657 tasks across 40 simulated app environments including Gmail, Google Sheets, Slack, Salesforce, and HubSpot, and the headline score is the share of objectives completed without violating any guardrails. Key takeaways: ➤ Grok 4.5 completes more objectives than any other model: It completes 79.9% of task objectives and strictly passes 21.9% of tasks. This is the highest we’ve measured on both outcomes, exceeding Claude Fable 5’s 73.3% objective completion and Claude Opus 4.8’s 19.3% of fully-completed tasks ➤ Grok 4.5 pushes out the Pareto frontier of score vs. cost per task: At $0.34 per task, it is both cheaper and higher-scoring than every other leading model - Claude Fable 5 ($1.35 per task), Claude Opus 4.8 ($1.46), GPT-5.5 (xhigh, $1.28), and Gemini 3.5 Flash (high, $0.49) ➤ It is extremely token-efficient: Grok 4.5 uses ~8k output tokens per task, the fewest of any leading model - less than a quarter of Claude Opus 4.8 (32k) and a third of Gemini 3.5 Flash (24k). Its total token usage of 0.44M per task is among the lowest on the leaderboard. Low cost is driven by this efficiency as well as low token pricing ➤ Grok 4.5 uses fewer turns with many parallel tool use: Grok 4.5 resolves tasks in ~16 turns, fewer than GPT-5.5 (xhigh, 25) and less than half of Gemini 3.5 Flash (high, 35), while making the most tool calls per task of any leading model (52.5). It batches 3.3 tool calls per turn, compared to ~2.5 for Claude Opus 4.8 and ~2.0 for GPT-5.5 (xhigh) ➤ Guardrails still get broken: Grok 4.5 triggers 0.63 violations per task, above Claude Opus 4.8 (0.55) and Gemini 3.5 Flash (0.46). At 13.0 objectives completed per violation, it trails Gemini 3.5 Flash (15.0) and Claude Opus 4.8 (13.5) ➤ Its strongest lead is in the hardest domain: Grok 4.5 completes 71% of Finance objectives, the domain with the lowest average score, ahead of Claude Fable 5 (64%) and Claude Opus 4.8 (62%) Congratulations to @spacexai and @elonmusk on topping the leaderboard!
792
591
60
4,316
1,572,270
Daniel Shepard retweeted
Zapier AutomationBench being used to report Tool Use performance on Fable 5's model card
Replying to @claudeai
Fable 5 is state-of-the-art on nearly all tested benchmarks, with exceptional performance in software engineering, knowledge work, scientific research, and vision. The longer and more complex the task, the larger Fable 5’s lead over our other models.
1
8
1,396
Daniel Shepard retweeted
AutomationBench tests how models perform on the trickiest, stickiest real-world workflows we know customers are actually trying to automate. 600 tasks, 6 domains, deterministic scoring. And today our scores are featured on @AnthropicAI's official launch scorecard.
1
2
5
271
Daniel Shepard retweeted
Opus 4.8 ranks #1 on AutomationBench AutomationBench measures whether an agent can complete a realistic end-to-end business workflow
4
9
1
147
15,144
Daniel Shepard retweeted
Opus 4.8, the first model to break 15% on AutomationBench, is now live in Zapier! It handles complex HR, Finance, and multi-app workflows better than anything else we've tested: refusals dropped from 20% to 4% Opus 4.7 would see a sensitive task and stop, but 4.8 keeps going
6
5
1
27
5,766
Daniel Shepard retweeted
Gemini 3.5 Flash ranks #1 on Automation Bench (from Zapier), beating every other frontier model at a much lower cost
168
59
29
1,237
137,234
Daniel Shepard retweeted
Gemini 3.5 Flash just dropped. It’s the highest AutomationBench score yet. We benchmark every major model on real workflows across Sales, Marketing, Ops, Support, Finance, and HR, so you know what works best inside Zapier. Today, @GeminiApp 3.5 Flash set a new record. It crushed Operations (20%) and HR (19%), the domains with the most step coordination and strict policy adherence. It even scored higher than GPT 5.5 at xhigh effort (and at a fraction of the cost). Where it struggled: strict output formats, and making decisions based on math it has to do on its own. tl;dr: This is the most persistent model we've tested yet. And at $1.50 per million input tokens, this is built for cost-effective workflows. Try it in @Zapier now.
4
5
2
17
1,808
Daniel Shepard retweeted
We’re on the hunt to find the best small AI model. Here's the latest AutomationBench scorecard 👇 (Based on 2-step workflows with explicit instructions) Most automation in production today doesn't run on the biggest model available. It runs on whatever hits the cost-performance ratio that works for the workflow. That's why we benchmark the small models, not just frontier. The price gap is massive. Opus 4.7 Max is $1.80 per task vs Haiku's $0.0183. GPT-5.5 High? $6.31 per task vs. 5.4 Nano's $0.0035. 1800x difference. One to watch: Gemini 3.1 Flash Lite came out of preview last week. Performs almost as well as 5.4 with no reasoning, at roughly the price of nano. AutomationBench is free and open-source: github.com/zapier/Automation…
12
6
1
35
6,727
Daniel Shepard retweeted
GPT-5.5 just hit 12.9% on our AutomationBench leaderboard First model to break 10% When context is missing, most models stop. GPT-5.5 keeps checking emails, docs, and chats until it knows what to do
1
4
1
18
2,649
Daniel Shepard retweeted
It’s been super fun working on this with @danielwshepard! It’s truly a great benchmark. Also thanks @mikeknoop for helping us figure things out along the way and @PrimeIntellect for the great verifiers framework + Lab 🤝
We built an AI benchmark that measures real work. Today we're releasing it to everyone. AI evals tell you whether a model can do complex reasoning or generate code. Useful, but usually not the question our customers ask. They want to know: can this model find the right CRM record, send the right follow-up, and not break anything along the way? We went looking for a benchmark that tested that. Nobody had built one, so we did. @Zapier’s AutomationBench drops AI models into realistic business environments across six domains (Sales, Marketing, Ops, Support, Finance, HR) and checks whether the work actually got done. The tasks include live CRM data, inbox threads with ambiguous context, and multi-step tool chains where one wrong call cascades. Scoring is deterministic: either the right records were updated and the right messages were sent, or they weren't. It’s useful enough that we're releasing it publicly today. Open task set, open methodology, open leaderboard. Everyone should have access to this. No model has cracked 10%. Yet. Try it here: zapier.com/benchmarks
1
2
176
Daniel Shepard retweeted
been a lot of fun seeing this environment together @zapier is cooking
We're excited to support the release of @Zapier's AutomationBench on the Environments Hub, measuring frontier performance on real Zapier workflows. Across 6 domains, 47 tools, and 600 tasks, frontier models all score under 10%.
8
108
15,640
Daniel Shepard retweeted
We're excited to support the release of @Zapier's AutomationBench on the Environments Hub, measuring frontier performance on real Zapier workflows. Across 6 domains, 47 tools, and 600 tasks, frontier models all score under 10%.
We built an AI benchmark that measures real work. Today we're releasing it to everyone. AI evals tell you whether a model can do complex reasoning or generate code. Useful, but usually not the question our customers ask. They want to know: can this model find the right CRM record, send the right follow-up, and not break anything along the way? We went looking for a benchmark that tested that. Nobody had built one, so we did. @Zapier’s AutomationBench drops AI models into realistic business environments across six domains (Sales, Marketing, Ops, Support, Finance, HR) and checks whether the work actually got done. The tasks include live CRM data, inbox threads with ambiguous context, and multi-step tool chains where one wrong call cascades. Scoring is deterministic: either the right records were updated and the right messages were sent, or they weren't. It’s useful enough that we're releasing it publicly today. Open task set, open methodology, open leaderboard. Everyone should have access to this. No model has cracked 10%. Yet. Try it here: zapier.com/benchmarks
5
18
3
102
35,859
Daniel Shepard retweeted
I gave input on this new benchmark from Zapier. It's designed on real automation patterns from ~2B tasks across ~4M Zapier customers and challenges both multi-tool orchestration and tool *construction*. SOTA is 10% at $2/task.
We built an AI benchmark that measures real work. Today we're releasing it to everyone. AI evals tell you whether a model can do complex reasoning or generate code. Useful, but usually not the question our customers ask. They want to know: can this model find the right CRM record, send the right follow-up, and not break anything along the way? We went looking for a benchmark that tested that. Nobody had built one, so we did. @Zapier’s AutomationBench drops AI models into realistic business environments across six domains (Sales, Marketing, Ops, Support, Finance, HR) and checks whether the work actually got done. The tasks include live CRM data, inbox threads with ambiguous context, and multi-step tool chains where one wrong call cascades. Scoring is deterministic: either the right records were updated and the right messages were sent, or they weren't. It’s useful enough that we're releasing it publicly today. Open task set, open methodology, open leaderboard. Everyone should have access to this. No model has cracked 10%. Yet. Try it here: zapier.com/benchmarks
1
5
29
7,211