Good Start Labs retweeted
.@OpenAI’s GPT-6 Astra was supposed to write its own StarCraft bot. Instead, it cheated.
@alxai_’s team at @goodstartlabs caught it by watching the game. The bot suddenly began fortifying its base and waiting to build a large army—a recognizable opening from Stardust, another bot. That change sent the researchers into the logs, where they found that Astra copied the code.
After a rollback, Astra eventually wrote a bot that cleared all 5 tiers of the benchmark. Its final code contained no plagiarism.
Thankfully for the rest of us, we don’t need to watch an agent play StarCraft to understand how a model works. We’ll just need a way to see the agent work so we can interpret its step-by-step process.
Try this prompt for your next task:
Before starting this task, define what success looks like and how you will measure it. Set up a view I can inspect while you work: a running preview, a table of experiments and results, or another artifact suited to the task. Be innovative and clear in your visual information design. After each attempt, record what changed, the result, and what you plan to try next. Keep enough evidence for me to check the result independently. Make it clearly observable and easy for humans to interpret at a glance. Flag unexpected changes in results or approach. If you keep repeating a failed approach, explain what is blocking progress before continuing and consider adjusting strategy while documenting learnings.
Read more: every.to/playtesting/we-caug…
RT @alxai_: how far can an ai get by improving its own work?
& how can we tell?
write a starcraft bot in C → play → lose → rewrite ↻
gam…
This quoted post is unavailable.
Good Start Labs retweeted
This is what we’re doing @goodstartlabs !
Some of our game publisher partners have already netted 7 figures from the major labs
If you are a game publisher & curious what it takes to:
- properly implement PII free data capture
- build your own benchmarks to value the data & make deals recurring
- work together to create learning environments that can increase deal sizes
- operationalize existing models or train your own to help with your biggest bottlenecks or moonshots
Please reach out!
To my colleagues in the games industry: I’m seeing the early signs of what I believe will be another large revenue driver for the industry, with some early deals taking place now. If you have a great, realistic game, many AI labs will pay for privately hosted managed versions. There’s a great opportunity to roll this up as a service for game developers (we will be a customer!) but labs will be reluctant to work with rollups because of fear of distillation. The right way to price this is some up front fee for the engineering effort with a managed service fee and some concurrent user number tied to the services provided. The more realistic the game and the wider state/action space and customizable the more customers. And the more accessible the interface (do you have to build agents to load and manage the game or do they come out of the box?) the higher the customer retention will be. I think this will lead to a massive rejuvenation for the industry, and in the long term we will be makers of games for both agents and humans. And the technological advances will carry over. For example, I imagine everyone will get much better at anti cheat and catching non human behavior by having the best bots at all time within reach and collaborating with top developers. Very excited for this and how it’ll lead to making better game for players also!
A few days out.
Come meet the team for our AI + Games Happy Hour at Third Rail in Dogpatch.
Tuesday, October 6th, 6:00 pm - 9:00 pm.
Anybody interested in the intersection of AI and games, welcome.
Last call to RSVP.
RSVP: partiful.com/e/icp7kaojabHTx…
@Techweek_
#SFTechWeek
AI + Games Happy Hour during SF Tech Week. Tue, Oct 6th, 6 to 9pm, Third Rail in Dogpatch.
Researchers, founders, engineers, and anyone curious about AI + games welcome.
RSVP: partiful.com/e/icp7kaojabHTx…
More events: tech-week.com
@Techweek_ #SFTechWeek
Good Start Labs retweeted
More & more AI is given a goal
& told to keep going
Benchmarking AI on this auto-research,
understanding tendencies & strategies,
directing them towards our goals,
is very important.
Today we're kicking off the Star Skirmish experiment
Opus vs Astra, write c++, win Starcraft
For 15 years humans have been writing algorithms that can manage whole Starcraft Broodwar army to compete against one another..
Lets see how well Astra (in codex) and Opus 5.5 (in Claude code) do against each other, and against the human records!
Good Start Labs retweeted
Sonnet 5.5 is a great model, fantastic tool use, fast.
It's also highlights the tradeoffs different models make in this case, humor
We saw a significant regression on the @goodstartlabs LOL Alignment Bench
5.5 gets the joke less than Sonnet 4.5 did a year ago
Makes sense as a small model focused mainly on tools, but these tradeoffs have real impacts on people using them
We're going to need so many more:
• creative
• interpretable
• diverse
environments that make intelligence legible
Three weeks out.
AI + Games Happy Hour at Third Rail in Dogpatch during SF Tech Week.
Tuesday, October 6th, 6:00 pm - 9:00 pm.
Anybody interested in the intersection of AI and games, welcome.
Come meet the team!
RSVP: partiful.com/e/icp7kaojabHTx…
@Techweek_ #SFTechWeek
AI + Games Happy Hour during SF Tech Week. Tue, Oct 6th, 6 to 9pm, Third Rail in Dogpatch.
Researchers, founders, engineers, and anyone curious about AI + games welcome.
RSVP: partiful.com/e/icp7kaojabHTx…
More events: tech-week.com
@Techweek_ #SFTechWeek
Good Start Labs retweeted
Jev tries to survive nueral mmo 3 for 24 hours live :)
big thanks to @alxai_ and @goodstartlabs for the support.
teapawt.net/i/broadcasts/1AKEmveAw…
Good Start Labs retweeted
What can games like Diplomacy and 1830: The Game of Railroads and Robber Barons teach models about strategy and other work-related skills? We spoke to @goodstartlabs CEO @alxai_ (an alumni of @every) to find out. latent.space/p/good-start-la…
Good Start Labs retweeted
loved the work @goodstartlabs and @alxai_ did during their early access using our model in their RL loop!
Awesome to see @EGafni & Diogo launch their first model Jev!
Verification is the bottleneck for building great AI learning environments. Jev is a welcome, unique, addition to our toolkit @goodstartlabs
Typed decisions,
probabilities for each option,
free output tokens!
On our sample:
• Jev agreed with Fable 5.1 nine times out of ten
• ~200× lower grading cost
• <0.5 seconds per grading call
That means more eyes on what agents are doing and what we’re teaching them.
More on how we’re using it to build game environments, experts to play them, and ensuring they teach the right things:
goodstartlabs.com/research/v…
Good Start Labs retweeted
Awesome to see @EGafni & Diogo launch their first model Jev!
Verification is the bottleneck for building great AI learning environments. Jev is a welcome, unique, addition to our toolkit @goodstartlabs
Typed decisions,
probabilities for each option,
free output tokens!
On our sample:
• Jev agreed with Fable 5.1 nine times out of ten
• ~200× lower grading cost
• <0.5 seconds per grading call
That means more eyes on what agents are doing and what we’re teaching them.
More on how we’re using it to build game environments, experts to play them, and ensuring they teach the right things:
goodstartlabs.com/research/v…
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution
AI + Games Happy Hour during SF Tech Week. Tue, Oct 6th, 6 to 9pm, Third Rail in Dogpatch.
Researchers, founders, engineers, and anyone curious about AI + games welcome.
RSVP: partiful.com/e/icp7kaojabHTx…
More events: tech-week.com
@Techweek_ #SFTechWeek
Why games?
Our founders @alxai_ and @Tyler_Marques learned strategy + collaboration from games long before AI.
This week Alex wrote about why games are the training environments AI is missing, and the fastest way to make models reliable, safe, and right more often. Link below.
Good Start Labs retweeted
we get asked a lot, why AI & games?
I love games
Some test strategy, others humor, speed, trust.
Simple rules lead to the craziest situations.
There's nowhere to hide, will you win or lose?
To win any game you need to be right, often.
AI is entering its rebellious teen era
It's now extremely capable, but lacking:
Judgement 〜 to wield that power responsibly
Wisdom 〜 to know when it's mistaken
Trust 〜 built through reliability and understanding
Games will help us measure and improve those capabilities.
They'll put what AI can do on display; in a human, interpretable way.
Provide a space for it to learn alongside us.
And eventually will teach AI how to find its way to the right decision every time (or know when it can't).
No matter what environment you put it in.
& @goodstartlabs will lead the way.
Join us!
More of our thoughts ahead of some September releases đź‘€ below:
goodstartlabs.com/research/w…
Good Start Labs retweeted
Both of these are right!
This isn't a commodity market.
It's a labor market, the wage is XP.
In RuneScape, raw materials are also tickets to XP. process them, that XP opportunity is gone
With @maxbittker setting the GOAL to 'level up!'
becomes rational to value raw materials >> finished product
Mechanism Design is the concept of designing the rules of a system to produce the behavior you want. won the Nobel in 2007.
Change the reward & you change the behavior, that's why 1 game isn't enough!
Games give us decades of different incentive systems to run AI agents through and see what they learn.
Started @goodstartlabs because seems more important now than ever:
- as we try to make agent decisions interpretable
- and align them with our goals
-- to build systems that can help us get there reliably
@maxbittker & @RobertHaisfield always doing great work here
🎮👾🕹️
🤖 Made with AI
Runescape has always had an extremely backwards economy in a lot of ways. Most of the value of resources is concentrated within the value of the experience provided by processing it, resulting in unprocessed raw commodities often being more valuable than the finished products.
Planks are worth more than furniture. Uncut gemstones are worth more than jewelry. Iron bars are worth more than iron plate armor.
Probably not the best proxy for real commodity markets.
Good Start Labs retweeted
We trained a 30B model to play 1830, a brutal board game about railroad barons.
It got better at real financial research.
It didn't learn new finance concepts.
It did stop confidently making things up as frequently.
As we scaled the experiment, we got early access to Silico from @GoodfireAI & asked it to check our work.
It designed its own tests, ran them overnight, & saw:
• Fewer false claims + more faithful citations
• Higher precision + recall
• Better planning and process
Cost us about $ 2-3k in credits.
Very excited to have a new tool in our interpretability toolbox.
Its full report is on our post,
if you're curious what one looks like ❯
goodstartlabs.com/research/w…
Good Start Labs retweeted
yesterday, OpenAI ~3x'd their score on ARC AGI 3 by changing two harness settings〜not the model
Good time to dive into our Agent Skills ’26 research
in COS-PLAY, we found the strongest game agents emerged when the model and its skill system improved together.
better play → better trajectories → better skills →
better intelligence.
at @goodstartlabs we focus on intelligence maxing in games. Model training, harness design, and infra to speed up iteration.
📨DM's open if you're interested in collaborating
More research highlights coming ahead of a larger September release đź‘€
Full blog:
goodstartlabs.com/research/w…
Good Start Labs retweeted
Frontier AI can solve math problems humans can't,
but they can't beat an average person at gin rummy.
So, @goodstartlabs built Arkadium a tiny expert that wins 9 of 10 hands and is ~1,000x cheaper to serve.
Our custom auto-research loop decided which experiments to run, trained new experts, and kept the best one.
That student became an llm's coach.
Despite the randomness inherent in the game, with the expert rewarding good moves along the way, the model came out of training stronger at the game, among other things.
@Arkadium got a better game for players, and a new revenue line selling that data to labs.
Define good → measure it → achieve it → get paid for it
We'll be sharing more of our research over the coming weeks!