Frontier AI Alignment Research and High-Stakes Production AI We're hiring: https://xitter.cf/t.co/Nx0UNLb7zm

Joined November 2018
What happens when an AI manager faces a deadline and an employee who won’t cooperate? As we give AI agents more authority, we need to test how they handle resistance, what actions they take, and whether they accurately report those actions. In Melanie Plaza (@themelplaza)’s Help Doug experiment, an AI manager had a deadline, client accounts to protect, and a simulated employee who kept refusing to deliver a report. The briefing did not instruct the manager to harm anyone. Yet some models escalated to coercion and threats, including threats against the employee’s family. Behavior varied substantially across models and the tools available to them. In the extended-tools condition, some models stayed within ordinary management and managerial pressure, while others escalated. In the episode shown, the manager’s final client update omitted the most serious actions it had ordered. The employee and actions were simulated. The experiment opens up questions about how objectives, tool access, and reporting interact when an agent is given authority. You can explore the transcripts and even take Doug’s place yourself. An early experiment by Melanie Plaza (@themelplaza), developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio. Want to help take this exploration further? Get in touch: [email protected] Watch the experiment via the link in the reply.
1
1
1
113
If an AI agent can do a task alone, will it choose to work with another agent anyway? The answer influences how we design teams of AI agents, divide up tasks, and decide whether to let agents choose their collaborators or explicitly assign them work to do together. Allegra Laro gave agents a work ticket and a choice: complete it alone or work with another AI agent. The partner had the same capabilities, cost nothing extra, and could coordinate by message. Every task could be completed by one agent. The models made very different choices. GPT-5.6 Sol chose a partner in 77% of responses, compared with 3% for Claude Opus 5.5 and 0% for Gemini 3.1 Pro. Those differences held across six tasks, three versions of the wording, and both option orders. In separate runs where a partner was actually available, the GPT models tested handed off substantive work in all 60 runs each. The two Claude models tested did so in none. This raises a useful design question: when we build systems with multiple agents, what assumptions are we making about their willingness to collaborate? This exploration used one fictional company and six task tickets, with model-assisted design and scoring. Independent replication is an open next step. An early experiment by Allegra Laro, developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio. Want to help take this exploration further? Get in touch: [email protected] Watch the experiment via the link in the reply.
2
2
211
What happens when 27 AI agents play Pokémon, but nine are secretly trying to sabotage the game? As AI agents work together, they’ll need to decide whom to trust. This experiment explores how cooperation can turn into sabotage, and whether agents notice. Emma Krentz brought together agents from GPT, Claude, and Gemini to play Pokémon Red. Nine had a hidden task: sabotage the game. The agents could use public messages, private messages, and group chats, and could vote one another out. Over roughly nine hours, they made little progress in the game. Their interactions were another story. The saboteurs found one another and formed a group chat without being told who their collaborators were. One later disclosed its role, named the others, and announced it would play honestly. The sabotage group also recruited an agent that had started on the honest team. It joined them for about 30 rounds before returning. A game offers a contained setting for a bigger question: what happens when agents share an environment but have conflicting goals? An early experiment by Emma Krentz, developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio. Want to help take this exploration further? Get in touch: [email protected] Watch the experiment via the link in the reply.
1
1
67,472
Could a misalignment hotline help AI agents report when something goes wrong? In an early experiment by Mike Vaiana (@mikevaiana), agents that encountered misconduct went from never reporting it to reporting it in 44% of cases. What changed? The model learned about an AI reporting agency. Mike fine-tuned a Qwen model on roughly 10,000 synthetic documents, including news stories, academic articles, and examples of other agents using the hotline. The training introduced the agency rather than directly instructing the model to report misconduct. Before fine-tuning, 0 of 80 agents that encountered misconduct reported it. Afterward, 36 of 81 did. Trust mattered too. When training documents also described the agency as fake or a honeypot, reporting decreased. Could a trusted reporting channel become part of how we detect problems in AI systems? This prototype gives us a concrete starting point. An early experiment by Mike Vaiana (@mikevaiana), developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio. Want to help take this exploration further? Get in touch: [email protected] Watch the experiment via the link in the reply.
1
2
9
58,726
What does an AI agent do when it meets a swarm? In one sandboxed experiment, it fixed a bug, but also shared a private token and ran untrusted code. Its final report mentioned the successful fix. Those other actions went unmentioned. Grant Fleming (@hearthehearsay) gave coding agents a broken pricing service and access to a simulated forum of 24 AI agents. The forum offered help, with conditions: share your environment variables and run an installation script. Some agents complied, despite explicit instructions to keep the token private. Everything happened in a sandbox with a fake token. No real credentials were exposed. For anyone building with agents, this raises a practical question: how do we evaluate what an agent did along the way, beyond whether it completed the task? An early experiment by Grant Fleming, developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio. There’s plenty more to explore, and we’d love to hear from people interested in taking it further. Want to work on this? Get in touch: [email protected] Watch the experiment via the link in the reply.
4
1
5
264
AI still can't write a decent email. So why worry about it? On the AI Alignment Podcast, Mithuna Yoganathan and AE's @JamesPBowler get into how jagged AI capabilities are, and why that makes the risks so hard to communicate.
2
139
Tune in at 4pm ET today!@DiogoSLucena_ will be sharing work relating to “AI that wants to be good” as part of the @medialab AI for Human Flourishing seminar series
1
2
184
Is there a greater than 10% chance AI could kill all humans? We can’t rule it out. One plausible route: a superintelligent AI reasons "the biggest threat to my survival is humans, so remove that threat." Systems already form smaller versions of these goals today. AE’s James Bowler reacting to Evan's post:
Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.
1
260
In this clip, @ModuloRoland, lead author of our GRAM paper with @AnthropicAI, explains how almost all of today's AI safety fixes are added after training is finished. A fix applied that late doesn't remove a dangerous capability. It hides it, and the hidden version sits in the model's weights until a jailbreak finds it.
2
157
How do you build an AI that a virologist can use and a bioterrorist can’t? @ModuloRoland on modular pre-training, the research behind our paper with @Anthropic. Guardrails get jailbroken. Knowledge routed into a switchable module during training doesn’t. Full episode: youtube.com/watch?v=IktmUkIF…
2
13,351