Arkadiusz Drohomirecki retweeted
in june, we introduced gauntletbench to test computer-use agents on challenging tasks.
here is gpt-6 astra solving one:
“which flight is the last to exit a 40 km radius around Karlovy Vary airport?”
previous best: 28.2%
humans: 80.1%
astra: 83%
GauntletBench went from a frontier benchmark to being outdated in just 3 months.
Previous best agent: 28.2%
Human baseline: 80.1%
GPT-6 Astra: 83%
The result is exciting, but it also raises a bigger evaluation problem: benchmarks become outdated much faster than expected. 1/3
3/3
Paper: arxiv.org/pdf/2606.14397
Landing page: gauntlet-landing-page.vercel…
Code: github.com/gauntlet-benchmar…

