IpezyGJ

Most impressive AI numbers don't survive a second look. I find the ones that don't — before they get published. My book is about why AI benchmark scores and trading backtests overpromise, and how to catch them. Pay what you want: leanpub.com/measurednotbelie…

A ranking with no sense of uncertainty is like a forecast that says "it will rain" with no chance attached. Confident, clean, and quietly hiding how sure it actually is.

Good measurement is boring on purpose: here's the number, here's how much to trust it, here's what would change it. The exciting version — one bold number, no caveats — is the one to doubt.

Same paired test, three benchmarks, public data: SWE-bench Verified — 129/133 adjacent ranks not separable MTEB — 176/180 HELM Lite — 89/89 (90 models, 10 scenarios) Boards separate distant systems fine. They can't order neighbours. github.com/ipezygj/swebench-…

"State of the art" is a headline, not a fact, until you see the gap to second place. If that gap is smaller than normal run-to-run luck, the headline is the only real thing there.

My job in one sentence: I re-derive impressive AI numbers from the source data before anyone bets on them. Some don't survive it. The interesting part is that some do.

I re-ran the math on the most-cited AI leaderboard myself, from its public votes. Its #1 actually holds up — backed by ~100,000 human comparisons. Sometimes the number is honest. You just have to check.

On the hardest coding benchmarks the leaders overlap once you draw the error bars. A 500-task set leaves a 96% score swinging about two points either way — more than the gaps being celebrated. The ranking looks solid. The very top of it usually isn't.

I checked a popular AI judge against its own scores: it picked the longer answer about two times out of three. That's not judging quality. That's rewarding length.

A benchmark you can't trust is worse than a lower one you can. A shaky 92% sends people the wrong way, confidently. A solid 70% at least points the right direction.

Two AI models "tied" at the top of a test often aren't equal — the test just ran out of ability to tell them apart. The fix isn't a bigger claim. It's a better test.

Imagine a judge who quietly prefers longer essays and its own writing style. Now make that judge an AI, grading other AIs. The bias doesn't vanish — it gets a leaderboard.

Grading quants by KL beats the usual vibes check, and it leaves one gap: at what KL does a task score actually move? A build can sit close in distribution and still lose the long chains where errors compound. Did they publish downstream accuracy per build, and on which set?

Every impressive AI number should come with a boring one next to it: how much it moves if you run it again. If nobody reports the wobble, usually nobody looked.

When a benchmark result sounds too clean, ask one simple thing: can I reproduce it from what they published? If the answer is "trust us," it's marketing, not measurement.