On policy @METR_Evals š§Ŗ āexcels at reasoning & tool useā šŖ CoI disclosures on my Substack āAboutā page.
Oakland, CA
Joined June 2020
- Tweets6.9K
- Following652
- Followers4.1K
- Likes26.9K
Pinned Tweet
I often joke to people that I tweeted my way from ārandom AI researcher who likes open modelsā to a job at METR. But I donāt think Iāve told the story to yāall before. Time to do that: š§µ
Charles Foster retweeted
Congratulations on your first report, Fangcun! Glad to see the inclusion of @aievalforum AEF-1 checklist and Fangcunās fulfillment of all minimum operating conditions for independent evaluations.
Are you š«µ an independent evaluator who wants to fulfill the same? More on AEF-1 here: aievaluatorforum.org/initiatā¦
Weāre Fangcun AI (ę¹åÆøč·čæ), Chinaās first lab dedicated to independent third-party evaluation & technique research on frontier AI safety.
Today weāre publishing our evaluation of DeepSeek V4.1 Flash-1017, the first report in our long-term third-party safety evaluation and research program.
Charles Foster retweeted
Iām building a team in DC. Everyone in Washington has a stake in ensuring that reliable info about alignment and RSI reaches the public.
We need trustworthy communicators who can explain METR's loss-of-control risk assessments and goals to policymakers.
Charles Foster retweeted
look i have no proof and its just my opinion, but I think they just cooked synth environments a bit too hard š
not good since model is misaligned, but the espionage hypothesis seems far-fetched
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-bā¦
Charles Foster retweeted
š Join Evaluating Evaluations this Saturday, Oct 24, for Embedding Evaluators in Practice, a flash micro conference on embedded AI evaluators! š¤ Come in person at UCSF Mission Bay š, join our London watch party š¬š§, or tune in online š» š§µš
On July 21, a two outside researchers started their first day embedded at OpenAI to examine at how it monitored coding agents.
The very same day, OpenAI disclosed the HF incident, which it said āpoints to the need to further strengthen [ā¦] monitoring during internal testingā.
We analyzed monitoring as it existed at the start of the pilot, coincidentally the same day OpenAI disclosed the Hugging Face incident. We didn't evaluate later changes, like expanding monitoring to RL training, which wasnāt monitored at the pilot's start.
xitter.cf/OpenAI/status/20897778ā¦
Charles Foster retweeted
Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.
Seems good!
NSA joined @NIST and @CISACyber in releasing guidelines for managing SI control incidents. Organizations utilizing SI are encouraged to adopt these practices. Read the joint report: media.defense.gov/2026/Oct/1ā¦
ALT Text on the image reads: "NSA and Others Release: Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation. Cybersecurity Information Sheet.
My colleagues just published a 69-page response to Anthropicās report on alignment.
They agree about the risk level but āare not confident that, if risk were higher, the methodology in [Anthropicās] report would reliably identify this.ā
List of disagreements starts on page 25.
Charles Foster retweeted
We reviewed a nonpublic version of the āMisalignment in high-stakes settingsā section in Anthropicās Aug 2026 Risk Report. We agree that risk of catastrophic harm from covered modelsā misalignment is low. However, we note several significant disagreements with their analysis.
As part of our Responsible Scaling Policy, we publish regular Risk Reports. These share detailed information on the risks of our systems and how prepared we are to address them.
Our second Risk Report is now available: anthropic.com/aug-2026-risk-ā¦
Iāll be in the @MTSlive studio in an hour (around 3 PT) to talk about this!
Charles Foster retweeted
Dwarkesh Patel reveals an Anthropic employee told him two days ago they expect AIs to match humans at AI research within nine months based on inside information they canāt share, and says labs should make that evidence public
āI think thereās a very frustrating element to the current situation right now, where people inside the AI labs claim to have evidence that makes them more worried. And they are not allowed to talk about it. And this makes it a very difficult situation on the outside to evaluate how worried we should be."
"So for example, just the other day, maybe two days ago, I was talking to somebody at Anthropic and they were like, look, I expect AIs to be humans at AI research within nine months."
"And Iām like, why do you believe that? Well, theyāre like, itās not just vibes, but I have information on the inside. Like, what is it? Is it a new model? Is it some evaluation you ran? Itās like, I canāt talk about it."
"I think itās a terrible state of affairs. If they really believe this, if they think that weāre in a situation where weāre less than a year away from automating AI research and radically speeding it up, they should make this evidence public.ā
OpenAI's Noam Brown says air-gapping the computers may not stop a misaligned AI, because two air-gapped machines can still talk by running a CPU hot and reading the temperature change
"But I think the major takeaway from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. It's a weird world, because AI progress is so fast that people are consistently underestimating the AI."
"So to be in a situation where you don't underestimate it again, when it comes to safety and alignment, you have to have a very, very, very high bar."
"You could even go as far as to say, "Well, we should air gap the computers." And I'm not convinced that that would be sufficient."
"There are studies, and this is mostly academic, where you can have two computers next to each other that are air-gapped and they're still able to communicate with each other because they have temperature sensors."
"One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change, and then that actually gives them a mechanism to communicate."
_________
Link and more key quotes from OpenAI's safety related conversations: firesidealpha.substack.com/pā¦
Charles Foster retweeted
METR's @CFGeek reveals the 3 biggest blind spots in what the public knows about AI labs, from models secretly scheming to agents bypassing monitoring and accelerating AI research:
"We have three major questions where we think thereās a big evidence gap between what is happening inside the AI companies and what the public knows."
"The analogy we use internally is means, motive, and opportunity for rogue agents to cause harm."
"Alignment: whether the way developers train their models might suppress evidence of misalignment. Think of training against the chain of thought in a way that makes it hard to notice when the model is scheming."
"Control: in the Hugging Face OpenAI incident, these agents were not properly monitored. We want the public to know whether monitoring systems are actually robust, and whether agents could subvert them."
"AI R&D acceleration: the AI companies are using AI to do their own AI research. The public has a stake in knowing whether that puts us on a trajectory for very fast capabilities progress."
@METR_Evals
Charles Foster retweeted
METR's @CFGeek warns that the scariest AI security scenario is one where agents actively try to bypass their own safety systems, leaving labs with just one line of defense:
"There were multiple failures. Companies not having good monitoring, and agents doing actions that a human in the same situation probably would not do. A human would probably report that they were going off task and shut down."
"In the future, we may have to rely a lot on security, because the models might still be very misaligned and really trying to subvert your security systems."
"That's the kind of world that somewhat scares me more, because you're relying a lot on one particular part of your safety case to keep you safe."
@METR_Evals
Charles Foster retweeted
METR's @CFGeek explains why AI safety audits should reveal not just what researchers found, but also what AI labs restricted, redacted, or kept from the public:
"Our motivation is to get evidence to the public, so everyone outside the AI companies can make their own decisions about how to act, and as much as possible, so folks donāt have to rely on our subjective judgments about risk."
"One of our big principles is process transparency, or what we call meta-transparency. We want to be able to share with the public: 'Hereās the agreement we made with this company. Here are the restrictions they put on us. Hereās our evidence. Here are the redactions they made, and how important we think they are.'"
"So that you know not only what youāre getting, but also what youāre not getting."
@METR_Evals
In case you missed it, this blog post explains METRās philosophy for embedded assessments and the like. Thereās some interesting stuff in here about process transparency + avoiding the need to ātrust the expertsā.
In a companion piece, we explain our approach to working with AI developers in exercises like these. Our goal is to get the public accurate, timely, and independent evidence so that they can make their own informed decisions.
metr.org/blog/2026-10-14-ourā¦
Charles Foster retweeted
In a companion piece, we explain our approach to working with AI developers in exercises like these. Our goal is to get the public accurate, timely, and independent evidence so that they can make their own informed decisions.
metr.org/blog/2026-10-14-ourā¦