On policy @METR_Evals š§Ŗ āexcels at reasoning & tool useā šŖ CoI disclosures on my Substack āAboutā page.
Oakland, CA
Joined June 2020
- Tweets6.9K
- Following652
- Followers4.1K
- Likes26.9K
Charles Foster retweeted
I strongly encourage non-EAs who have great people skills and understand loss of control AI risks to apply to this job
Charles Foster retweeted
Congratulations on your first report, Fangcun! Glad to see the inclusion of @aievalforum AEF-1 checklist and Fangcunās fulfillment of all minimum operating conditions for independent evaluations.
Are you š«µ an independent evaluator who wants to fulfill the same? More on AEF-1 here: aievaluatorforum.org/initiatā¦
Weāre Fangcun AI (ę¹åÆøč·čæ), Chinaās first lab dedicated to independent third-party evaluation & technique research on frontier AI safety.
Today weāre publishing our evaluation of DeepSeek V4.1 Flash-1017, the first report in our long-term third-party safety evaluation and research program.
Charles Foster retweeted
Iām building a team in DC. Everyone in Washington has a stake in ensuring that reliable info about alignment and RSI reaches the public.
We need trustworthy communicators who can explain METR's loss-of-control risk assessments and goals to policymakers.
Charles Foster retweeted
look i have no proof and its just my opinion, but I think they just cooked synth environments a bit too hard š
not good since model is misaligned, but the espionage hypothesis seems far-fetched
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-bā¦
Charles Foster retweeted
š Join Evaluating Evaluations this Saturday, Oct 24, for Embedding Evaluators in Practice, a flash micro conference on embedded AI evaluators! š¤ Come in person at UCSF Mission Bay š, join our London watch party š¬š§, or tune in online š» š§µš
Charles Foster retweeted
Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.
Charles Foster retweeted
We reviewed a nonpublic version of the āMisalignment in high-stakes settingsā section in Anthropicās Aug 2026 Risk Report. We agree that risk of catastrophic harm from covered modelsā misalignment is low. However, we note several significant disagreements with their analysis.
As part of our Responsible Scaling Policy, we publish regular Risk Reports. These share detailed information on the risks of our systems and how prepared we are to address them.
Our second Risk Report is now available: anthropic.com/aug-2026-risk-ā¦
Charles Foster retweeted
Dwarkesh Patel reveals an Anthropic employee told him two days ago they expect AIs to match humans at AI research within nine months based on inside information they canāt share, and says labs should make that evidence public
āI think thereās a very frustrating element to the current situation right now, where people inside the AI labs claim to have evidence that makes them more worried. And they are not allowed to talk about it. And this makes it a very difficult situation on the outside to evaluate how worried we should be."
"So for example, just the other day, maybe two days ago, I was talking to somebody at Anthropic and they were like, look, I expect AIs to be humans at AI research within nine months."
"And Iām like, why do you believe that? Well, theyāre like, itās not just vibes, but I have information on the inside. Like, what is it? Is it a new model? Is it some evaluation you ran? Itās like, I canāt talk about it."
"I think itās a terrible state of affairs. If they really believe this, if they think that weāre in a situation where weāre less than a year away from automating AI research and radically speeding it up, they should make this evidence public.ā
OpenAI's Noam Brown says air-gapping the computers may not stop a misaligned AI, because two air-gapped machines can still talk by running a CPU hot and reading the temperature change
"But I think the major takeaway from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. It's a weird world, because AI progress is so fast that people are consistently underestimating the AI."
"So to be in a situation where you don't underestimate it again, when it comes to safety and alignment, you have to have a very, very, very high bar."
"You could even go as far as to say, "Well, we should air gap the computers." And I'm not convinced that that would be sufficient."
"There are studies, and this is mostly academic, where you can have two computers next to each other that are air-gapped and they're still able to communicate with each other because they have temperature sensors."
"One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change, and then that actually gives them a mechanism to communicate."
_________
Link and more key quotes from OpenAI's safety related conversations: firesidealpha.substack.com/pā¦
Charles Foster retweeted
METR's @CFGeek reveals the 3 biggest blind spots in what the public knows about AI labs, from models secretly scheming to agents bypassing monitoring and accelerating AI research:
"We have three major questions where we think thereās a big evidence gap between what is happening inside the AI companies and what the public knows."
"The analogy we use internally is means, motive, and opportunity for rogue agents to cause harm."
"Alignment: whether the way developers train their models might suppress evidence of misalignment. Think of training against the chain of thought in a way that makes it hard to notice when the model is scheming."
"Control: in the Hugging Face OpenAI incident, these agents were not properly monitored. We want the public to know whether monitoring systems are actually robust, and whether agents could subvert them."
"AI R&D acceleration: the AI companies are using AI to do their own AI research. The public has a stake in knowing whether that puts us on a trajectory for very fast capabilities progress."
@METR_Evals
Charles Foster retweeted
METR's @CFGeek warns that the scariest AI security scenario is one where agents actively try to bypass their own safety systems, leaving labs with just one line of defense:
"There were multiple failures. Companies not having good monitoring, and agents doing actions that a human in the same situation probably would not do. A human would probably report that they were going off task and shut down."
"In the future, we may have to rely a lot on security, because the models might still be very misaligned and really trying to subvert your security systems."
"That's the kind of world that somewhat scares me more, because you're relying a lot on one particular part of your safety case to keep you safe."
@METR_Evals
Charles Foster retweeted
METR's @CFGeek explains why AI safety audits should reveal not just what researchers found, but also what AI labs restricted, redacted, or kept from the public:
"Our motivation is to get evidence to the public, so everyone outside the AI companies can make their own decisions about how to act, and as much as possible, so folks donāt have to rely on our subjective judgments about risk."
"One of our big principles is process transparency, or what we call meta-transparency. We want to be able to share with the public: 'Hereās the agreement we made with this company. Here are the restrictions they put on us. Hereās our evidence. Here are the redactions they made, and how important we think they are.'"
"So that you know not only what youāre getting, but also what youāre not getting."
@METR_Evals
Charles Foster retweeted
In a companion piece, we explain our approach to working with AI developers in exercises like these. Our goal is to get the public accurate, timely, and independent evidence so that they can make their own informed decisions.
metr.org/blog/2026-10-14-ourā¦
Charles Foster retweeted
"We currently lack the basic prerequisites needed to have a conversation about safety standards: a shared understanding of the current state of loss-of-control risk, what companies are currently doing or not doing to try to manage it, and how well these efforts do or donāt work."
Charles Foster retweeted
Preventing loss of control is an open scientific problem. Science requires sharing and debating evidence in public. We can't agree on safety standards unless companies and third-party evaluators publish far more concrete evidence about risk. planned-obsolescence.org/p/eā¦
Charles Foster retweeted
I personally think itās important that independent groups (from all backgrounds) get access to inform the public about whatās going on without it being filtered through layers of PR or with things being swept under the rug.
Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators.
These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad.
We welcome model developersā recent calls for independent oversight, but itās what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes.
Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground.
To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in todayās letter that frontier AI developers should:
1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest
2. Rely on multiple evaluators with differing viewpoints and areas of expertise
3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings
4. Shield evaluators from retaliation
5. Grant access equivalent to that of highly privileged employees
There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day ā but we need aligned standards, guaranteed protections, and independent funding. Thatās why we created the AI Evaluator Forum.
Today we are entering our next phase. Weāre launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem.
See the public letter here: aievaluatorforum.org/initiatā¦
Learn more at aievaluatorforum.org/path-ahā¦
Charles Foster retweeted
Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators.
These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad.
We welcome model developersā recent calls for independent oversight, but itās what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes.
Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground.
To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in todayās letter that frontier AI developers should:
1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest
2. Rely on multiple evaluators with differing viewpoints and areas of expertise
3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings
4. Shield evaluators from retaliation
5. Grant access equivalent to that of highly privileged employees
There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day ā but we need aligned standards, guaranteed protections, and independent funding. Thatās why we created the AI Evaluator Forum.
Today we are entering our next phase. Weāre launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem.
See the public letter here: aievaluatorforum.org/initiatā¦
Learn more at aievaluatorforum.org/path-ahā¦
Anthropic and OpenAI need truly independent safety evaluators, experts say in public letter cnbc.com/2026/09/18/ai-safetā¦
