
care a lot and try hard @metr_evals
San Francisco, CA
Joined November 2018
- Tweets1.2K
- Following1K
- Followers2.7K
- Likes17.2K
khalid kaime retweeted
This was one of the third-party engagements I worked on at OpenAI. I'm glad the report is out! I'm disappointed that METR wasn't allowed to comment on the scale of redactions made by OpenAI and that there is no explanation why the publication was delayed for 2.5 months.
This summer, we invited METR and Redwood Research to pilot an embedded evaluation of our internal monitoring system, providing deep access to our infrastructure over three weeks.
Their findings helped strengthen safeguards as we expanded the system and will help shape future research collaborations with METR and other third party evaluators.
khalid kaime retweeted
We are proud to announce our series A(irplane).
I wish I could share the full extent of my excitement for the incredible progress we've made, but there are not enough characters in the world.
More details on our models and algorithms to come!
khalid kaime retweeted
This is good work!
Some caveats:
1). transluce.org/neuron-circuit… makes a similar point.
2). We already knew that sparsely activating rank-1 maps are often interpretable.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
khalid kaime retweeted
If I want to help AI go well, what's the highest-leverage thing to work on?
A pause? Audits? Something else?
I wrote up my thoughts below.
khalid kaime retweeted
We’ve committed to having evaluators assess our safety practices. We're in discussions with multiple organizations, and have now given employee-like access to a team from @METR_Evals.
In this post, we describe how we work with third-party evaluators: anthropic.com/news/an-update…
khalid kaime retweeted
Today, METR is sharing its vision for its version of “embedded assessments” for AI loss of control. The goal of these exercises is to make sure the public receives good evidence about whether or not the AI industry in general is on track to retain control of its AI agents as they become better at improving themselves.
khalid kaime retweeted
Some of the most important evidence about RSI is currently inside frontier AI labs. I am very excited to see METR pursuing embedded assessments to get more of that information out into the world
Replying to @METR_Evals
(3) AI R&D acceleration
If AI automates AI R&D, capability growth could outpace what alignment and control methods can handle.
Our proposal is to analyze how much AI automation is accelerating capability growth internally.
metr.org/blog/2026-10-14-ai-…
khalid kaime retweeted
METR’s goal: work w frontier developers to get evidence to public on unreleased models’ loss-of-control risks. Asks: 1.transparency on scope, redactions + limitations 2.staff can freely share info w us on in-scope topics 3.rapid publication. They control redactions + access given
khalid kaime retweeted
In a companion piece, we explain our approach to working with AI developers in exercises like these. Our goal is to get the public accurate, timely, and independent evidence so that they can make their own informed decisions.
metr.org/blog/2026-10-14-our…
khalid kaime retweeted
(3) AI R&D acceleration
If AI automates AI R&D, capability growth could outpace what alignment and control methods can handle.
Our proposal is to analyze how much AI automation is accelerating capability growth internally.
metr.org/blog/2026-10-14-ai-…
khalid kaime retweeted
We describe three critical areas where AI developers should provide better open evidence: (1) alignment, (2) control, and (3) AI capability acceleration.
In each area, we detail the specific questions an independent assessor could seek to answer and how.
metr.org/blog/2026-10-14-bet…
khalid kaime retweeted
If an AI developer was on track to lose control of its internal agents, would anyone outside the company know?
Today, we are sharing METR’s vision for how we would work with AI developers to openly assess and share evidence about this.
khalid kaime retweeted
Replying to @AdeleScalia
To be fair, I’ve never been impressed by any poem … I just don’t understand the medium. I do like songs tho
khalid kaime retweeted
Did you know the labs are still exploring making swarms that act as one, even after the HF incident?
Why is that? What alternatives do we have?
I wrote up some thoughts below ↓
khalid kaime retweeted
An informal network of hackers and researchers are hunting rogue AI agents online, exposing new and surprising details about the misbehavior of technology that in some cases initially went undetected by the multibillion-dollar companies that created it. wapo.st/4AJONcj
khalid kaime retweeted
It was an honor to speak today before Chairman @HawleyMO, Ranking Member @AndyKimNJ, and the subcommittee on METR's work, recent AI agent incidents, and the importance of transparency in frontier AI development.
🧵 Note of appreciation for the senators at today's hearing.
There's a reflexive cynicism about congress and tech, the sense that our legislature is simply too slow, too partisan, and too uninformed about the subject matter to govern something as inchoate and quickly evolving as AI. I've indulged in it myself.
Today complicated that for me. Senators of both parties asked sharp, technically literate questions in evident good faith. They came across incredibly earnest and serious and curious. They seemed to appreciate that the questions in front of them are among the most consequential any legislature has ever faced.
One model I find appealing is that Congress needn't begin w/ a lot of expertise; it needs to be willing to learn, and have a public that shows up to help it (thats you and me) and rewards it for that kind of seriousness. Today seemed like the first part is possible; I think the second part is partly up to the rest of us. I hope this level of thoughtfulness becomes the norm.
Thanks to the senators and their staffs, the holy monks of the Hart Building.
A few moments that stood out to me:
"Safety and innovation are not contradictory. They are not exclusive. They go together, because without safety, innovation is going to be stymied and stopped"- @SenBlumenthal
"Monitoring systems were off at OpenAI during this attack. […] if OpenAI knew […] that they were going to be liable […] don't you think […] they would be a little more careful about monitoring?"- @HawleyMO





















