Pinned Tweet
Launching Game Gym 🔥
> 11 AAA games
> 5-hour time limit per game
> vision-based (no game state, no SDK, …)
Check out evals.nunu.ai to see how well models can play games!
Introducing GameGym: The ultimate gaming benchmark for AI agents 🎮
Today we're releasing parts of our internal evaluation environments to track frontier model performance on video games.
It measures two main KPIs: completion rate and speed. Similar to speedrunning, while completion rates may eventually saturate, speed is designed to be an evergreen metric for continuously measuring improvements.
Games are amongst the hardest challenges for AI to master: 3D, real-time, long-horizon, non-deterministic and massively diverse environments.
We believe it is the perfect benchmark built for the endgame.
Check out early results here:
evals.nunu.ai/benchmarks/gam…
1
10
502
Astra 6.2 gets 59.3% in Game Gym🤯
Game Gym measures the models performance on 11 AAA game titles (Dark Souls, Trackmania, Polybridge, ...)
Some hightlighs below 🧵
Conspiracy: OpenAI's thinking traces have become so abstract that their summarizer has problems making sense of them.
The thinking summary writes about CREATING a tutorial. The actual task is about PLAYING an existing tutorial of a game.
All actions and other tool calls make perfect sense for playing the tutorial, just the thinking summary doesn't match.
Todays model are reliably perform long-horizon tasks.
It doesn't matter too much anymore if you are using e.g. Astra vs Sol vs Terra if the task doesn't have a difficult step.
In the KSP task of Game Gym you can see that terra is just as good as sol
1
13
407
Still far from perfect but...
Watch an LLM natively play Hogwarts Legacy.
> Fully vision based
> Raw Keyboard/Mouse actions as output
2
5
120
Gamdev is indeed changing. It is kind of poetic that GTA 6 releases in the year where you can almost one-shot prompt it into existence.
A historic milestone that will separate two eras.
AI is making games faster and cheaper to build, giving developers more room to take creative risks, revive overlooked genres, and let players reshape the worlds they play in.
In this conversation, a16z's Jon Lai and Anish Acharya join Elena Burger to discuss why human taste still determines what makes a game fun, how modding created some of gaming's biggest franchises, the opportunity for new distribution platforms, and games that adapt to whatever players choose to do.
0:00 Intro
0:57 How AI can now build a game
6:21 What shorter dev cycles unlock
10:08 Can AI make a game fun?
12:27 When creating is the game
14:01 Building a Habbo Hotel record shop
16:16 His son rebuilt Ghost of Tsushima in Minecraft
17:54 Mods: where billion-dollar franchises come from
20:36 Decompiling and reviving dead games
22:45 The renaissance of niche genres
25:51 Managing AI agents like an RTS game
28:02 The business of AI games
33:53 What excites and scares studios
35:16 The never-ending game
37:56 The games they'd finally buQild
YouTube: youtu.be/HDRy6XmE2RQ
@VirtualElena @illscience @Tocelot
1
1
58
Hydrogen is pretty good at Polybridge 🔥
Fable 5.2 is now our best performing model on APK arena.
Slowly saturated as well. Phone-use is basically solved at this point 🔥
1
2
48
We had early access to Gemini 4 Carbon 🔥
A lot of our early testing got botched by terrible inference times (up to 5mins for a tep).
But the model itself is actually good. I expect a much higher score on our internal evals once inference is fixed.
Gemini 4 Carbon is now available.
It’s our strongest model, built to dig deeper, handle long-horizon tasks, and multi-step reasoning.
If Argon is great for complex tasks, Carbon is the model you bring in for your hardest problems.
7
2,037
About 3 years ago I had a very long discussion with @jjschnyder on how AI will play games in a few years.
Jan predicted models will be able to write a "Godscript" while I claimed they will predict the next keyboard/mouse actions in high frequency.
Today, it looks like both are true. Astra is presssing WASD with pausing the games between moves, while Opus wrote a whole Script to solve the level.














