AI Safety Research Programs @ConstellOrg, Anthropic Fellows, OpenAI Safety Fellowship, ex-MATS Apologetically myself
Berkeley
Joined March 2022
- Tweets1.1K
- Following790
- Followers2.2K
- Likes10.3K
Pinned Tweet
Memento mori for the modern age.
Props to @summeryue0 for calling this out!
Henry retweeted
I'm just speculating like everyone else (no juicy secrets about this shared with me) but here's my guess about what is going on right now with OpenAI and the three fired employees:
The negotiations between OpenAI and METR are ongoing. OpenAI has said they want embedded third parties, METR has proven their worth with the previous engagements, so METR is an obvious choice. But my guess is that METR is asking for more this time; probably they want more access, more time on site, less narrow scope, the right to comment on redactions, etc.
(Compare with the terms of their previous engagements: In this one xitter.cf/METR_Evals/status/2113… a third of their report was redacted and they weren’t allowed to say why, and in the Hugging Face investigation they only had 6 days and were only allowed to investigate the precursor swarms nor the subsequent more severe compromise of OpenAI infrastructure. xitter.cf/DKokotajlo/status/2092…)
Various researchers at OpenAI probably support METR in this (including, I’d imagine, the fired three employees? Tomek was the point person on the hugging face investigation), but also various higher-ups at OpenAI and legal types probably oppose and want more restricted access and scope – basically they want to work with an org that’ll go easy on them and not give them trouble.
OpenAI holds most of the cards here, since this whole thing is voluntary and not legally required, and they can threaten to go with Accenture or HuggingFace or whatever instead. Anyhow to improve their bargaining position and/or minimize the blowback if they decide to say no to METR and go with someone more pliable, it helps if they can discredit METR.
Hence (I suspect) why they are going after my friend Ryan Greenblatt (who works at METR) now. They don't need to present actual evidence, they just need to cast vague aspersions, and many employees will give them the benefit of the doubt. If and when they say "We chose to work with a different third party" they'll face lower risk of employee discontent than if METR's reputation was unsullied.
This whole mess, by the way, was predictable. *Of course* there's a big political fight about who should be trusted to do this and who has relevant expertise and independence etc. and *of course* the companies are trying to make things shake out in a way that favors them.
It's part of why I've been advocating for *actually pacing the frontier* instead of just doing third party assessments. "We allocate at most 20% of our compute to AI R&D" is a lot easier to verify than "we are being safe and responsible" and depends a lot less on subjective judgment and technical alignment expertise -- AND, if you do it, people will be able to see the slowdown in the trends of agentic coding performance in the frontier companies and see laggard companies start to catch up. Harder for it to end up just being safetywashing / industry capture.
Based on this article, OpenAI has made false accusations about me. The three researchers fired from OpenAI were told I facilitated architecture leaks to The Information. This is false.
A journalist from The Information did reach out to me shortly before that article came out. I did not comment at all beyond asking the journalist to make it clear that a blog post I wrote which was referenced in the article was preexisting and public.
The article also mentions an internal OpenAI monitorability proposal. I have heard that this memo contained highly sensitive architecture information, and that OpenAI accused me of accessing and reading it. This accusation is also false.
As reported in the article, I viewed a version of the proposal that did not have sensitive IP and was approved for sharing with me. I believe OpenAI now knows this accusation was based on a conflation of these two documents, though I do not know if they have clearly retracted that accusation.
I do not and have never solicited proprietary information, and the three employees who were recently fired from OpenAI never leaked IP to me. As the article mentions, I did have Signal chats with them that had disappearing messages on; they’re professional contacts that I’ve had on Signal for a while, and use of Signal is common in the tech industry. During the Hugging Face investigation, I was careful to avoid sending Signal messages with confidential information, though I did sometimes discuss high level things on Signal calls. As far as I understood, this was normal.
Henry retweeted
In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: Huggingface.wtf
Henry retweeted
Lots of ppl have criticised Anthropic's bill for making it too hard for the govt to shut them down
This thread defends Anthropic: routing shutdown through the courts is needed to avoid extreme power concentration.
Idk how to strike the tradeoff.
I do wish AI safety spent more time asking "what would happen if the executive abused my proposal to take control of AI development"
Henry retweeted
seeing a lot of this going around lately
Maybe I shouldn't even say this out loud. But it seems to be that if one is an EA doomer, the principle and belief consistent action is to us AI to undertake a controlled but still scary hacking incident in order to show the world the risk.
Clearly no amount of Eliezer Yudkoswky essays are going to convince the world of the magnitude of the risk. So why not do something like a false flag cybersecurity incident?
Sure you will go to jail, but if the world is probably ending you probably won't even see a cell. And aren't they focused on taking actions with high public expected value? This seems like exactly that sort of thing.
Obviously there are some EA doomers making their best contribution to risk reduction in other ways. But what could be higher expected value than waking society up to the risk six months early? Surely there are some willing to sacrifice themselves for this?
Again, this is my saying what seems like follows from their beliefs, not mine.
Henry retweeted
I've spent a bit more time with Nick's work. I feel cautiously optimistic about these results, but I don't want to get my hopes up before seeing more concrete demos. Regardless, I am very very glad that someone is pursuing this research direction!
Some miscellaneous thoughts and takes:
- The main thing I would want to see is how Nick's interpretability insight could be applied to existing problems such as:
= Understanding model counterfactuals (see e.g., alignment.anthropic.com/2026…)
= Search for unexpected model behaviors (e.g., transluce.org/weirdchat)
I bring this up not because I don't believe in fundamental interp research, but because I no longer trust people's intuitions around whether something qualifies as actually advancing understanding. Being able to do something others cannot with your interp techniques is sort of a "proof of work."
(I basically agree with everything Sam Marks wrote here, which makes this same point: lesswrong.com/posts/wGRnzCFc…)
- When I think about how to advance ambitious interp, superposition isn't something that's very prominent. We want some sort of sparse decomposition regardless, since human attention is limited and we can only try to focus on so many things at the same time. I think a meaningful advancement would be if we can draw these attribution graphs that are actually accurate, such that (1) we can do counterfactual interventions on the graphs, and these result in predictable results and (2) these graphs can allow us to do various downstream tasks, such as the ones I just mentioned above.
(This is not the only way to advance ambitious interp, but it's the one that I'm most interested in.)
- The narrative around this essay is structured on Nick's personal optimism and excitement, and how he come to hold these optimistic views. On the one hand, I think it's good for authors to have the freedom to share their vision with the world in an unconstrained way. Unfortunately, I think the tone has resulted in people drawing way too rosy of a conclusion than what is warranted. Many others have rightly pushed back on this publicly.
(And also come on, look at the LLM results, do they seem like do you that we've "solved interp?")
- On the other hand, man, I just want some people to do attribution graphs on high-ish level concepts and try to understand model cognition. I feel like there hasn't been any major work on this on actually large LLMs since the Biology paper (Anthropic seems to have stopped putting out work in this direction? And the parameter decomposition work from Goodfire earlier this year was irrc on small models.) Just the fact that Nick is giving this a go by itself is imo commendable. I think we do in fact want to throw a whole bunch of AIs at this problem, so it'll be good for people to work on this now. (e.g., it's good that Nick has tried to use agents to understand neurons today, because we'd really want to do get good at that before its too late.)
- Corollary: My guess is that we can in fact learn a lot from working with higher level concepts like "Cookies, milk" to interpret LLM cognition. I think the lower level workings of LLMs—how they understand grammar or code syntax, for example—is probably a lot messier. In other words, I think it's probably a lot easier to understand high level LLM behaviors like "when would models reward hack?" using interpretability, than to understand how GPT-2 XL achieved some level of loss on the pertaining dataset.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Henry retweeted
After reading for 15 mins: I think Nick is operating in a world where interpretability was just about enumerating the neurons and superposition was an obstacle to that plan, so now he's happy because maybe superposition isn't real. But:
1) there are v few empirical results. Unfortunately interp has a long history of significantly overclaiming how interpretable their findings are (SAEs, early circuits having poor causal scrubbing scores etc) so I'm not updating much at all on the evidence he's already shown. it's really got a long history of people getting overexcited. So looking forward to seeing the paper with all the empirical evidence!
2) I think it's not that big news for me even if true because superposition in something like the average case wasn't my main crux to thinking interpretability is tractable. some cruxes this doesn't really deal with at all:
- I think the correct 'circuit' is usually just an overwhelmingly large bag of random-seeming heuristics (a la Redwood Research’s interpretation of how an LLM predicts is vs was) and having it isn't very informative/doesn't help with much/doesn't lead to good predictions.
- More generally, It seems possible to enumerate all the 'low level' features in a neural network, and still have a hard time answering 'high level' questions like: "Is the model misaligned? / What would it do in this new situation? / If it is misaligned what should we do about it?" there's an abstraction/emergence gap to cross from the features to the questions of interest, and crossing that gap is likely much harder than enumerating features.
That said, if his claims hold up, I think it will be quite a big deal, and a useful precursor to tackling the thornier problems! There may also be some useful things to do without tackling the thorny problems: eg we'll be able to make use of masses of automated agents to scour the features and circuits to try and make sense of the mess, and maybe that’s good for something. although I suspect that even the most optimistic outcome here will not make alignment of superhuman AIs much easier (as some on twitter have claimed) because the neuron interpretability will not be adversarially robust, among other issues
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Henry retweeted
And this is of course all worsened by how secrecy corrodes reasoning, as Ellsberg once taught Kissinger:
“you’re about to get a lot of clearances higher than top secret that you did not know existed. That’s going to have (...) effects on you. First, a great exhilaration that you’re getting all this amazing information that you didn’t know even existed.
And (...) you’ll feel like a fool for not having known of any of this. But that won’t last long. Very soon, you’ll come to think that everyone else is foolish. What would this expert be telling me if he knew what I knew?” So in the end, you stopped listening to them.”
Dwarkesh Patel reveals an Anthropic employee told him two days ago they expect AIs to match humans at AI research within nine months based on inside information they can’t share, and says labs should make that evidence public
“I think there’s a very frustrating element to the current situation right now, where people inside the AI labs claim to have evidence that makes them more worried. And they are not allowed to talk about it. And this makes it a very difficult situation on the outside to evaluate how worried we should be."
"So for example, just the other day, maybe two days ago, I was talking to somebody at Anthropic and they were like, look, I expect AIs to be humans at AI research within nine months."
"And I’m like, why do you believe that? Well, they’re like, it’s not just vibes, but I have information on the inside. Like, what is it? Is it a new model? Is it some evaluation you ran? It’s like, I can’t talk about it."
"I think it’s a terrible state of affairs. If they really believe this, if they think that we’re in a situation where we’re less than a year away from automating AI research and radically speeding it up, they should make this evidence public.”
Henry retweeted
I'm pretty excited about this as a prototype for risk assessments, but it's important to remember that this is not a substitute for formal oversight. METR is an assessor working under a voluntary agreement, not an auditor (because there are no set safety standards to audit).
We’ve committed to having evaluators assess our safety practices. We're in discussions with multiple organizations, and have now given employee-like access to a team from @METR_Evals.
In this post, we describe how we work with third-party evaluators: anthropic.com/news/an-update…
Henry retweeted
The most important technology in history is outpacing our ability to understand it.
We’re funding exceptional talent to work on it.
Henry retweeted
We’ve committed to having evaluators assess our safety practices. We're in discussions with multiple organizations, and have now given employee-like access to a team from @METR_Evals.
In this post, we describe how we work with third-party evaluators: anthropic.com/news/an-update…
Apply now to Generator!
Generator is the best way to jump in and get involved in AI Safety if you're looking to do anything other than pure research.
Excited to be mentoring again! So impressed by the residents, their projects - and how quickly they're all getting hired :))
50+% of participants getting hired before the end of a program is disgustingly good.
On this metric, Generator beats every other fellowship in AI Safety (which ime usually reach ~80% lifetime hiring rates after a year or so).
Apply now!
Henry retweeted
METR’s goal: work w frontier developers to get evidence to public on unreleased models’ loss-of-control risks. Asks: 1.transparency on scope, redactions + limitations 2.staff can freely share info w us on in-scope topics 3.rapid publication. They control redactions + access given
Henry retweeted
You shouldn’t have to trust METR because AI companies should just publish risk-relevant data themselves. Until then, our job is to make important evidence open so that the public can come to their own conclusions.
We are ready to work with frontier labs to make this happen.
Henry retweeted
I can't really see a world where OpenAI's actions are justified here.
It was Tomek's job to communicate with METR, who were being given significant, unprecedented access to do the HuggingFace report. The norms were not set. Even if he fucked up badly, just give him a stern warning and ban him from roles involving liasoning with third parties, he did lots of other valuable work, and there'd no longer be room for him to inappropriately leak as part of his job.
IMO firing people over a good faith attempt to use their best judgement in a novel and uncertain situation is a sign of a highly unhealthy culture. Firing him will have a chilling effect on anyone else at OpenAI working with third parties, an odd choice when Sam announced they'd embed evaluators with high access.
I separately consider this a tragedy, the METR report was one of the most important works of safety this year, punishing Tomek for being a part of it, and making it harder for future such reports to exist is a real shame.
Of course, I've only heard Tomek's side, but unless he's outright lying I'm struggling to see ways this looks good for OpenAI?
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
Henry retweeted
> violated clear policies
> won't say what the policies are
> “not about raising safety concerns”
> doesn’t deny firing him for talking to third-party safety orgs
> we prioritize monitorability
> deploys and profits from model that destroys monitorability trends
subpoena plz
A note from our research leaders:
Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment. We generally keep individual employment matters private and don't believe a back and forth would be productive or lead to a resolution, but we want to address the points they raised in their letter directly.
- We want to be very clear that these decisions were not about raising safety concerns or speaking out. Safety and research debates happen every day at OpenAI, often spirited and highly critical. We actively encourage these discussions and consider them essential to making the right decisions. We cannot do the work in front of us without a high degree of trust. We will continue to be extremely forgiving of our team making good-faith mistakes. We have not and do not terminate any of our employees for raising concerns.
- We are actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks. People across the company have been working really hard on getting these partnerships up and running. We are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work. Many of our researchers already work with 3p safety organizations productively.
- We agree with the letter that preserving the monitorability of frontier models requires an industry-wide commitment, including from OpenAI. Monitorability has long been a core piece of our research program, and something we continue to invest significant resources in (see our publications on Monitoring Monitorability and the subsequent open sourcing of monitorability evals, our system card for GPT-6 Astra, Jakub’s blog and post on X, and the numerous blog posts on our Alignment blog on the topic).
We are deeply sad about this outcome. We appreciated Jasmine, Mikita, and Tomek’s contributions to AI safety at OpenAI and their willingness to speak up and challenge ideas. We championed their voices, supported their work, and placed enormous trust in them. These decisions were not about them raising safety concerns. We have always encouraged that and always will.
Henry retweeted
New paper.
Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning).
New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!
Henry retweeted
OpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear:
