We've long known that AI agent benchmarks are broken. Check out our work back from July 2025 on establishing best practices for making rigorous AI agent benchmarks! (links below)

New research from Bridgewater AIA Labs and UIUC, in collaboration with Thinking Machines! We use @tinkerapi to achieve state-of-the-art on a cleaned version of the BIRD text-to-SQL dataset, outperforming all frontier models we tested (including Fable and Sol) up to 8x cheaper. We achieve a proxy for human-level accuracy with no external scaffolding and a single model (w/ multiple rollouts). The common view on fine-tuning is it reduces price. Our results show the feasibility of achieving a higher performance at a fraction of the cost on economically valuable tasks, when incorporating task expertise into task-specific training. 1/6

Prior work has shown a counter-intuitive idea that RLVR boosts reasoning using 100% noisy data to a similar level of using clean data. We show that this is false through a more rigorous process of constructing noisy data. We demonstrate that RLVR with noisy data leads to worse MATH reasoning than RLVR with format-only rewards and over 9% lower accuracy than RLVR with clean data. This is a gap that hasn’t been resolved by existing algorithmic improvements. 1/9

Building text-to-SQL agents? Don’t blindly trust benchmarks/leaderboards. In our VLDB 2026 paper, we find >50% annotation error rates in BIRD and Spider 2.0-Snow, two widely used text-to-SQL benchmarks, shifting agents’ measured performance by up to 19%. 1/9