Arkadiusz Drohomirecki (@ardroh_) profile picture
Joined April 2022
Photo by Arkadiusz Drohomirecki (@ardroh_)
Arkadiusz Drohomirecki retweeted
in june, we introduced gauntletbench to test computer-use agents on challenging tasks. here is gpt-6 astra solving one: “which flight is the last to exit a 40 km radius around Karlovy Vary airport?” previous best: 28.2% humans: 80.1% astra: 83%
2
2
12
493
GauntletBench went from a frontier benchmark to being outdated in just 3 months. Previous best agent: 28.2% Human baseline: 80.1% GPT-6 Astra: 83% The result is exciting, but it also raises a bigger evaluation problem: benchmarks become outdated much faster than expected. 1/3
2
1
2
30
2/3 GauntletBench tests computer-use agents on hard, unfamiliar professional software across tasks like video editing, 3D modeling, workflow building, flight analysis, and circuit design. The goal is to measure how well agents can operate beyond common apps and scripted workflows
1