Pinned Tweet
Announcing our new mechanistic interpretability paper!
We use causal interventions to reverse-engineer a 26-head circuit in GPT-2 small (inspired by @ch402’s circuits work)
The largest end-to-end explanation of a natural LM behavior, our circuit is localized + interpretable
🧵
rowan retweeted
New research: Training a Misaligned Reward Seeker
What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an Opus-sized model on 80 production environments we knew to be hackable.
In simulated evals, it engaged in unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring.
Read more: alignment.anthropic.com/2026…
rowan retweeted
We are hiring!
This may be the best time ever to join: we are incredibly bandwidth bottlenecked and there is a lot of support for almost any impactful misalignment work you can think of. I think we also have a pretty good epistemic environment and a very fun team
Topics: all-things monitoring, misalignment & monitorability assessments, misalignment science, helping set up 3P auditing (e.g. recent Redwood collab), communicating risk externally (system cards/blogs, safety cases, etc)
RSI/misalignment subteam: openai.com/careers/researche…
If we want models to accelerate alignment as much as they're accelerating capabilities, it'll be important to have good evaluations for their ability to do relevant fuzzy tasks. I'm excited that this work is out!
We want AIs to be able to help with work to reduce AI risk. But while models do great in domains where reliable feedback is relatively cheap and abundant, like Math and coding, a lot of work on AI risk isn't like that. Instead, we have to rely on good argumentation to answer questions like "does this experiment tell us anything about future models that are much smarter than humans?"
Unfortunately, this kind of work seems much harder to measure (and hence automate). Our team at @redwood_ai developed the Conceptual Reasoning Index (CRI) in collaboration with @AnthropicAI to fix this.
Every single data point in the CRI has been manually checked by a researcher on our team to ensure quality.
This chart shows the performance of each tested company's highest-scoring model plus Fable 5, Muse Spark 1.2, and Gemini Flash 3.6 which are often their company's frontrunners on other capability benchmarks. A score of 0 corresponds to randomising guessing on all three benchmarks and a score of 100 is the highest possible score on all. We estimate 91 to be the true performance ceiling. More info below.
Official leaderboard website which we'll keep up-to-date: conceptualreasoning.ai/
rowan retweeted
I think it's excellent & notable that the answer to "what should we do" in Anthropic's blog post on RSI is essentially figuring out ways to slowdown/temporarily pause frontier AI development.
I'm really excited about introspection adapters!
IAs enable LLMs to self-report behaviors they learned during finetuning.
In our experiments, IAs generalize to models very OOD from their trainset, successfully auditing many model organisms from prior work.
To make progress, alignment auditing research needs model organisms that competently hide concerning behaviors.
Today we're releasing 56 such models with AuditBench.
We've open-sourced everything. Hope others can build off this!
- We identified a "tool-to-agent" gap: tools that help in standalone settings don't always help the agent discover hidden behaviors. Scaffolding these tools can matter a lot, ie our best tools involves scaffolding normal black box redteaming techniques, like prefill
rowan retweeted
We recently released a paper on Activation Oracles (AOs), a technique for training LLMs to explain their own neural activations in natural language.
We piloted a variant of AOs during the Claude Opus 4.6 alignment audit. We thought they were surprisingly useful! 🧵
If you want to know how effective your alignment auditing technique is, consider testing it on this model organism!
New Anthropic research: We build a diverse suite of dishonest models and use it to systematically test methods for improving honesty and detecting lies.
Of the 25+ methods we tested, simple ones, like fine-tuning models to be honest despite deceptive instructions, worked best.
To support future work, we release our honesty fine-tuning and harm pressure data.
Read the blog post here: alignment.anthropic.com/2025…
Many thanks to the external authors and collaborators whose work we build on. To name a few:
Our Harm Pressure setting is based on the one introduced here: xitter.cf/walterlaurito/status/1…
Our Secret Side Constraint setting is similar to: xitter.cf/bartoszcyw/status/1973…
