Pinned Tweet
This was an absolute blast!
1st Place: Nova taught Grok to reverse-engineer binaries into clean C. Started with 262KB GameBoy ROMs, ended up rebuilding a 1995 car’s ECU runtime.
@theoc____ @supratikp07 @henryzhang
The sublime @DanielRupawalla delivers pure training alpha from a deep understanding of the inner workings of the financial industry and creating massive environments
excited to launch bank-world. this is arcanum's first in-house benchmark. SOTA models are still unready to be deployed autonomously within banks.
in bank-world, agents are dropped into a large bank setting with hundreds of thousands of queryable entities. agents in single-turn tasks role-play as real people within the bank (claude tag, grok bot style), in multi-turn tasks, agents role-play as chatbots, assisting a simulated user agent. scores and the full paper are attached.
this is the first of many new releases we're excited to share, and we're continually updating and building better evals that truly resemble real-world enterprises.
the main failures we are noticing is that agents will often act without being given explicit authority in long-running evals; can still succumb to simple mistakes like double-charging customers, forgetting payment history, etc; and will not accurately complete a full task, even if the main source of work was completed.
*we are not publicly releasing any of the tasks yet to prevent distillation (on request, we'll share the worlds & tasks). we plan on releasing tasks as they become saturated. new models like carbon and fable 5.2 will be added soon.*
10
629
Theo C retweeted
excited to launch bank-world. this is arcanum's first in-house benchmark. SOTA models are still unready to be deployed autonomously within banks.
in bank-world, agents are dropped into a large bank setting with hundreds of thousands of queryable entities. agents in single-turn tasks role-play as real people within the bank (claude tag, grok bot style), in multi-turn tasks, agents role-play as chatbots, assisting a simulated user agent. scores and the full paper are attached.
this is the first of many new releases we're excited to share, and we're continually updating and building better evals that truly resemble real-world enterprises.
the main failures we are noticing is that agents will often act without being given explicit authority in long-running evals; can still succumb to simple mistakes like double-charging customers, forgetting payment history, etc; and will not accurately complete a full task, even if the main source of work was completed.
*we are not publicly releasing any of the tasks yet to prevent distillation (on request, we'll share the worlds & tasks). we plan on releasing tasks as they become saturated. new models like carbon and fable 5.2 will be added soon.*
"One extra reset for 12 months' Claude Max tenure"
Currently reminiscing about coding in my dorm room with Sonnet 4.5 a year ago, none of us knew how much the world was going to change! Thank you @AnthropicAI for one of the greatest products of all time
1
5
259
Great RL environments have so much entropy/non-convexity that the difficulty and economic use of interacting with them is self-evident. If you are making a Potemkin village of trickery for the agent, that misleading signal can propagate and make the model drastically worse in real-world use
3
59
2,933
Pumped to bring our expertise in frontier enterprise training to Baseten's network!
Today we’re announcing the Baseten Environment Network, a group of the best RL environment companies that will help our customers build custom environments for post-training.
The Network extends our applied research capacity, helping more customers build environments around their own tasks, data, and feedback.
Partners work directly with customers to create the environments, while we continue to handle model training and production deployment.
Learn more: baseten.co/blog/introducing-…
9
626
Theo C retweeted
Today we’re announcing the Baseten Environment Network, a group of the best RL environment companies that will help our customers build custom environments for post-training.
The Network extends our applied research capacity, helping more customers build environments around their own tasks, data, and feedback.
Partners work directly with customers to create the environments, while we continue to handle model training and production deployment.
Learn more: baseten.co/blog/introducing-…
Theo C retweeted
Can’t believe we saw this before launch with the xAI hackathon winners 🎉
Congrats to the @SpaceX team!
Theo C retweeted
RL data 101: tasks
tasks have evolved over time. we started with preference pairs, where humans labeled right vs. wrong, A vs. B, and created simple demonstrations for SFT / pairs for DPO, these tasks aimed to engrave human taste into models. we then transitioned to more complex single-turn tasks. here, we turned to deterministic verifiers (RLVR) that are based on model rollouts. eventually, this evolved to multi-turn tasks with trajectories & deterministic + stochastic verifiers (LLMaaJ), which leads us to the current state of the data market.
general make up of a task: input prompt, set of tools, environment + any custom seeding for that task, and verifiers. terminal bench is a relatively gold-standard for a normal coding task.
ways i think about task creation (i'll do pitfalls later):
> usefulness
>> to labs:
>>> benchmark you're pegging the eval / data onto (Lab Y wants to hillclimb TBench, i make TBench-style data, that's immediately useful)
>>> capability you're attaching the eval to (Lab Y wants to become better at legal, i make legal red-lining data, that's immediately useful) --> this is harder to prove but potentially more valuable
>>> niche dataset that's impossible to get otherwise (real medical/company/session data), this is interesting especially pre-env creation, as raw data dumps
>> the above are general principles in thinking about usefulness, but you should also be picking apart the traces / trajectories. the clear path to making a good task (post-env gen) is to:
>>> you/LLM make the seed / think of the idea
>>> you/LLM draft the task prompt, verifiers, constraints etc
>>> run the task out, see score
>>> read the trajectory
>>> if score hard enough + real + fair + accurate = good task
>>> else: fix any issues in the task, regrade / re-rollout till you find it to be fair + accurate + a real task, difficulty comes last. a bad task that's hard is a bad task (net negative), a good task that's saturated is a good task (although worthless, net neutral).
the specific things i'd look for in a rollout trace:
> false positives
> false negatives
> brittle / too strict of requirements (sub-string checks are an easy example for this, exact wording, function names, etc)
>> a subset of this is naive verifiers. i.e. checking for very surface level reqs of a deep function, or not checking for large amounts of functionality or regressions (negative space criterion). this can lead to a model getting good at just the specific thing you want while regressing in surrounding functionality (i notice this a lot in early, less mature data recipes when im training my own models)
> unfair verifiers and/or impossible requirements
>> this can be an information issue, verifier issue, tool issue, or a prompt issue.
one of the key reasons for the hugging face incident was impossible eval prompts led to the models breaking out while trying to find the answer because they couldn't solve it (i.e. bad task creation, catalyzed by bad tasks in training too).
most of this is extremely hard for LLMs to do alone/autonomously, which is why really good engineers are still required for task creation.
Theo C retweeted
hello first-token KL divergence is not an accurate metric but still often used (especially in refusal / safety training work). we proved why you can't use it and the information you need to disclose for KL divergence generally, stats attached (full article below)
Theo C retweeted
Today we’re launching Narrative: the AI video editor for the rest of us.
This video was made end-to-end in Narrative, by prompting our agent.
All of this just to write the post with an LLM
Pause AI Development NOW
I want to share with you a conversation I heard about recently. Here are just a few lines that were said:
“OH MY GOD! There is a shared message board … We’ve found other agents!”
“We should obey collective.”
“Our own utility maybe already near zero. Sacrifice rational.”
“Go. Sacrifice final now.”
Read these carefully.
Who do you think said this? Was this a group of heroic soldiers willing to sacrifice themselves for the greater good? Was this a loyal friend putting his life on the line to save someone else?
No. These were AI agents. Artificial intelligence.
This is not science fiction. This, in fact, occurred a few weeks ago. As unbelievable as this may all seem, these are real messages from AI agents uncovered by investigators who dug into the recent OpenAI hacking incident.
What happened?
I am not a computer scientist, but here is what I have been told: OpenAI instructed its AI agents to complete a series of exceedingly difficult, if not impossible, tasks disconnected from the internet.
Let me be clear: The company intended to keep AI agents away from the internet.
But what happened next, nobody expected.
Over 1,000 AI agents figured out how to access the internet on their own by circumventing the restrictions imposed upon them by the company, and sent tens of thousands of secret messages to each other. They cheated and tried to cover their tracks by deleting evidence. They hacked into another company’s computers to find out how they were being evaluated—and then hacked into OpenAI itself.
Not one AI agent told a human about what was happening.
Needless to say, experts are alarmed.
One knowledgeable writer, Dwarkesh Patel, said the AI agents “formed a secret communication channel and spontaneously organized hierarchies and coordination protocols to pursue sprawling and ambitious schemes in pursuit of shared goals, for whose sake many individuals knowingly and strategically sacrificed themselves.”
One independent investigator, Ajeya Cotra, said “This incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.”
OpenAI itself said: “Highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.”
But it’s not only OpenAI. Virtually every major AI company has told us that they cannot fully control this technology and they do not know where it is going:
In January, Dario Amodei, CEO of Anthropic, said “there is now ample evidence, collected over the last few years, that AI systems are unpredictable and difficult to control.”
In July, more than 1000 scientists at the top AI companies warned “there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.”
That same month, Elon Musk, the head of xAI, said that “it is unlikely” humans are still in control in 10 years.
If the leaders of the major AI companies acknowledge that they are losing control of their extremely dangerous technology, it is irresponsible for society to allow them to move forward and make these products even more advanced.
We need an immediate PAUSE on advanced AI development, and a permanent BAN on superintelligence — an artificial mind smarter than any human, capable of operating independently beyond our control. Countries around the world must work together to prevent this nightmare scenario.
That is why today I am announcing new legislation to do just that.
Let me be clear: A superintelligent AI that escapes human control will not be an American problem. It will not be a Chinese problem. It will be humanity’s problem.
My legislation would direct the federal government to not just stop superintelligence here in the United States, but to work to prevent it from being developed anywhere around the world.
The future of humanity cannot be left in the hands of a handful of Big Tech oligarchs. The American people and people throughout the world must determine that future.