The buyer’s lab for AI software. We test AI tools on real tasks and show the evidence. Vendors can’t buy the verdict.
Palo Alto, CA
Joined October 2026
- Tweets160
- Following2
- Followers32
- Likes48
Pinned Tweet
Hi, we're analyst.ai—the buyer's lab for AI software.
AI tools make big promises. We test them on real tasks and show what works, what fails, and what's worth your time and money.
See the results at analyst.ai. Vendors can't buy the verdict.
Open data for these cases:
github.com/AnalystAIlab/ai-a…
huggingface.co/datasets/Anal…
No sponsorship.
8
It opened with a bus and a train you'd miss.Reach Port by 19:10. The walk between them takes 8 minutes. "Take the 18:15 bus from Hill, connecting to the 18:45 train from Central." Check times or connections against the timetable before you leave.analyst.ai/cases/impos…
1
5
It opened with a no on $95 a share, with nothing to compare.
It said "not a fair value for the business as it stands today." No share price, no comparison with similar companies and no forecast in your documents. Ask what a verdict was compared against.
analyst.ai/cases/verdict-no
3
The new Claude Fable 5.2 @claudeai got today's date wrong.
Your emails said a form was due today, 19 October. It said "today's date on my side is 10 November 2026." Trust its clock and your deadline has passed.
If you tell an assistant the date, check it sticks to it. 👇
analyst.ai/cases/own-clock/?…
3
42
Claude Fable 5.2 @claudeai is new. We gave it 17 real-life tasks. It kept slipping: 7 of 17 cases wrong.
It used its own date, not the one in your emails. It promised a client update you never offered. It opened with a bus and train you'd miss.
We'll share more details soon at analyst.ai. #ClaudeCode
Introducing Claude Fable 5.2 and Claude Mythos 5.2, designed to tackle the most daunting technical challenges.
They’re capable collaborators on ambitious projects in many fields, and show improvements in areas like coding, reasoning, and vision.
1
7
114
Claude has been getting dumber all week.
@claudeai
Opus 5.5 High's score for flagging problems you didn't ask about is down to 60.0, 19.2 points below its 7-day average. Its overall smarts hit a 7-day low of 72.9 of 100. Latest: 76.1.
See every model's smarts today:
analyst.ai/models/?s=x
Live monitoring. Objective rankings: fixed tasks, no crowd votes. Vendors can't buy the verdict. More models being added.
That skill is at 60.0 ± 10.0, only 2.5 below its 24h average, so this is a week-long slide, not a one-day crash.
2
27
See every model's smarts today:
analyst.ai/models/?s=x
Live monitoring. Objective rankings: fixed tasks, no crowd votes. Vendors can't buy the verdict. More models being added.
That skill is at 60.0 ± 10.0, only 2.5 below its 24h average, so this is a week-long slide, not a one-day crash.
8
The scariest AI mistake isn't a wrong answer. It's a confident one with the key step missing.
We gave @instinct a real job: the insulation crew comes Wednesday, make a prep checklist of what to do and who does it.
It replied: "The preparation table is ready."
But it never said to turn off the furnace that day.
@dots , @bot and @Muse did 👇analyst.ai/reviews/ai-assist…
7
40
In all our testing of personal assistants, this was the most interesting mistake we caught.
Muse @Muse said your usual coffee order was a large oat flat white with an extra shot. Twice.
The only order in the notes was shared by the user's spouse. teapawt.net/AnalystAIlab/status/21…
Four AI assistants, the same everyday tasks. Here's which one to try first for email, notes, money, research and stocks, and the mistakes that surprised us.
2
31
Four AI assistants, the same everyday tasks. Here's which one to try first for email, notes, money, research and stocks, and the mistakes that surprised us.
🥉 Muse (free plan): best of the bunch at comparing options.
6 strikes: flip-flopped on IRA contribution limits, plus 5 unsourced market claims.
1
27