Adam Edgerton retweeted
In my opinion any AI graphs intended for wide public dissemination should not use log scales.
This graph looks so chill but what its saying is actually extremely not chill (to the point it was a big factor in convincing Jacob to leave Anthropic and raise the alarm).
This slide from the Anthropic IPO roadshow was a big factor in my decision to leave. It’s an insane plot and I’m surprised they made it public.
It shows the compute efficiency of pretraining over time. The y-axis is a **log scale**: they are seeing continued superexponential algorithmic progress.
Mythos was not a one-time deviation, but a point on a superexponential. They could already retrain Mythos with a fraction of the compute originally needed.
It doesn’t strictly imply a software-only intelligence explosion- they have been spending more on R+D. But algorithmic improvements do not require ridiculous creative genius and are on the verge of being automated. You do the math.
Adam Edgerton retweeted
Behind the blockbuster news this week, there is something happening quietly in the background with major implications for AI governance:
On Monday, California will announce how it plans to build out the first-ever IVO framework.
It’s clear that “independent evaluation” as it stands today is not structured to be truly independent. There are no rules for conflicts of interest, working with the labs is by invitation only, and labs determine the scope and access.
Today, a lab and an evaluator operating with the best of intentions are working within a structure that holds evaluators accountable to the labs, not the public.
California’s law changes all that; the IVO framework flips that paradigm.
For the first time ever, third-party experts evaluating AI models and systems will be held accountable to the government and the public, not the labs, for their work.
The government will determine independence and expertise. They will also be able to hold IVOs accountable to their outcomes. We’ll be able to reliably monitor leading (not just lagging) indicators for catastrophic risk, with an enforcement mechanism.
We designed and championed IVOs because they marry the immediate need for AI oversight with the immediate capabilities we have to do it, while setting a foundation we can scale, adapt, and build on.
But the details for how we build this matter, and they’ll determine if we create a race-to-the-top marketplace of rigorous evaluators, or end up with captured entities that form an ineffectual rubber stamp.
Here’s what @Fathom_org recommended to California as they navigate the tough design choices that come from building this in practice for the very first time:
(1) Outcome Standards: Require independent evaluation for catastrophic risk for frontier labs through an amendment to SB53. California should set the outcome standards, defining an acceptable level of catastrophic risk mitigation, and let IVOs determine whether or not it was achieved.
(2) IVO Payment and Selection: Take a phased approach: at first, the state should assign IVOs to labs from a pre-designated pool, with developers paying on a time-and-material basis. As more IVOs get designated, developers can choose among them, while the state retains the power to veto any selection, mandate IVO rotation, select additional IVOs to check others’ work, and decertify an IVO at any stage.
(3) IVO Readiness and Market Maturity: California should sequence the IVO mandate rather than stall implementation to wait for supply.
(4) Access for IVOs: The state should set the contours of every engagement. Today the IVO and the developer negotiate scope and access privately. The state should publish a standard engagement framework and resolve disputes when the two sides disagree.
(5) Recursive Self-Improvement: IVOs should be mandated to monitor the risk of AI systems that can improve themselves in both internal and external use. Embedded evaluations will be critical for tracking the share and speed of work done through RSI within the labs, verifying that labs are preventing unsafe internal model deployment, and set capability thresholds with deployers that would trigger notification to the state and a hold internal deployment until the IVO re-assesses.
(6) Enforcement: The Attorney General can issue an injunction to halt development, internal use or deployment of an AI model. Emergency powers allow the administration or the AG to do the same in the case of imminent catastrophic risk. IVOs can identify risks and recommend corrective actions, but the relevant state authorities decide when and how to take action.
Adam Edgerton retweeted
Did you read the recent METR report and think, “What is going on?!!! And how do I do something about it?”
Come work with us! The CNAS AI Policy Lab is hiring. We’re looking for extraordinary individuals who are passionate about getting AI policy right. cnas.org/careers
Adam Edgerton retweeted
Big day for Fathom. We are announcing our Executive Council, which will help guide our strategic decisions. Some of them have been with us since the early days. I am honestly humbled that this group has chosen to give us their time, reputations, and efforts.
Bringing about meaningful AI oversight that works at the speed of this technology is a generational challenge that will take a big-tent approach, across sectors, partisan divides, and even international boundaries.
This council is made up of former governors of both parties, national security leaders and AI experts who know how complex policy needs to work in practice. Excited to have them on board.
fathom.org/about#executiveco…
Adam Edgerton retweeted
“A big-name push for AI audits”: @BrendanBordelon has the first look at our new Executive Council in @politico Decoded this morning.
“Fathom was the driving force behind California’s passage this year of SB 813, which requires the state to set standards for outside groups that evaluate AI safety risks. It now wants to spread that to other states, countries and Washington DC.”
Adam Edgerton retweeted
We’re growing, and we’re looking for someone to help lead our work in Washington.
@fathom_org is hiring a Vice President of Federal Policy. The next two years are a big opportunity for federal AI policy, and this person will help keep thoughtful, effective ideas at the center of the conversation.
The role owns and executes Fathom’s federal agenda: setting our legislative and regulatory priorities, building bipartisan relationships across both chambers, and working with Members, committee staff, and agencies to move good policy forward. They’ll also be Fathom’s main voice in DC, at convenings, with the press, and with coalition partners, industry, and other stakeholders.
We’re looking for someone with deep relationships across Washington who can put them to work from day one, and who’s known as a trusted authority rather than a partisan tactician.
Together, we’ll keep building Fathom’s federal policy program into one of the most trusted in AI governance.
If that sounds like you, or someone you know, we’d love to hear from you. ⬇️
Adam Edgerton retweeted
One of the things I'm most encouraged by in this work is what becomes possible when philanthropy, technical expertise, and public-interest leadership come together around a shared challenge.
Adam Edgerton retweeted
22 leaders from 20 countries just backed a call for independent evaluation of frontier AI, "with qualified evaluators granted sufficient access to assess risks."
Australia, Canada, Singapore and the EU were four of the six geographies in our readiness report ⬇️
Adam Edgerton retweeted
Some big news today on independent verification of AI systems: Governor Newsom just signed an executive order that accelerates the implementation of IVOs in California.
At @Fathom_org we’ve been pushing for IVOs for years, and today we are seeing a state turn what was once a policy idea into a real life regulatory structure to keep people safe as this technology advances.
BREAKING: @CAGovernor Gavin Newsom just issued an EXECUTIVE ORDER directing actions to address the dangers of recent AI incidents.
The order accelerates California’s new law establishing first-in-the-nation independent oversight of AI companies and safety checks and advances the creation of an “AI kill switch.”
More to come — watch this space!
Adam Edgerton retweeted
BREAKING: @CAGovernor Gavin Newsom just issued an EXECUTIVE ORDER directing actions to address the dangers of recent AI incidents.
The order accelerates California’s new law establishing first-in-the-nation independent oversight of AI companies and safety checks and advances the creation of an “AI kill switch.”
More to come — watch this space!
Adam Edgerton retweeted
So we actually built this!
Paper: arxiv.org/abs/2606.31567
Demo: ai-reports.org/
We are actually looking for an org to run this fulltime as researchers in their individual capacity don't have bandwidth. If you are an independent org interested in maintaining a registry of flaw/incident reports and follow up with model providers pls reach out!
i think we need to create a "model card" equivalent for reporting misalignment incidents, this would guarantee a certain level of transparency and help build a better understanding over time. some ideas for what the fields could be:
- date of the incident/detection/report (already present in the examples oai reported!)
- frequency: how often does this behavior happen (number or % of rollouts affected)
- stage: does this happen during eval or RL training. if training: do we expect this behavior to be reinforced by RL? evolution of % of rollouts affected over time
- detection: was this incident caught by the current monitoring system?
- task category: broad description of the tasks where the misalignment happened (cyber, research, web search, basic Q&A, biology etc.)
- model family: what model family is affected (Sol, Astra etc.)
- novelty: is this an issue we were already aware of or not?
- external impact: did the incident have an external impact (i.e. wiki incident would have been yes)
this is just some random ideas i had (more in thread that are a bit more "complex"), we need to add more that would contribute to increase transparency and understanding. but it's also very important that this does NOT slow down the process of reporting misalignment behavior!
some examples from the incident reported by oai recently
Adam Edgerton retweeted
Very excited to publish the Transformative AI strategy for Europe.
After months of thinking, drafting and negotiating, I feel there is a path: if we do all of this, Europe can still thrive.
@privitera_ and @MonikaSchnitzer assembled a great effort, and I was happy to contribute three recommendations:
- European countries should seek compute-for-access deals with American AI developers to trade datacenter sites for assured access to the frontier.
- Europe and like-minded middle powers should forge an alliance to coordinate on supply chain bottlenecks and resist coercion.
- Europe should either actually spend a trillion Euros to build frontier AI or not, but stop the sovereignty theatre.
More fundamentally, I believe Europe taking AGI seriously must mean rebuilding the transatlantic alliance, grounded in trust backed up by mutual commitments and hard leverage.
Very few strategies have captured that aspiration before, but I believe this document strikes the balance very well.
I have many more thoughts on the role of middle power alliances and 'allied scale' in that future coming up, but for now: let's start implementing the Transformative AI Strategy today.
Today, we publish a Transformative AI Strategy for Europe.
Over the last few months, we’ve rallied researchers and engaged with governments to develop a plan for protecting the prosperity, sovereignty, and security of Europeans in a time of rapid AI progress.
transformative-ai.eu
@MonikaSchnitzer and I are honored to have convened an all-star team of thinkers and researchers contributing ambitious near-term objectives to make Europe relevant again, including:
— Creating a Member State Alliance for Supply Chain Security (@anton_d_leicht et al)
— Making European institutions ready to act in a transformative AI world (Conor McGlynn et al)
— Securing Europe's share of global AI compute, in a ‘European Way’ that benefits local communities (@philip_fox_ et al)
— Ensuring resilience to AI crises (@ben_s_bucknall et al)
— Making Europe the global leader in assurance technology
— And more objectives around security of supply and leverage (@milorignell et al), economic strength (@FraukeStehr et al), and safety/security (@NoemiDreksler et al)
Our all-star senior expert council of Europe’s best and brightest (and non-European friends) reviewed drafts, provided strategic advice, and made suggestions for how to make the strategy more useful: @Ph_Aghion @Christophkw @bakkermichiel @ischinger @vestager @aleks_madry @FuestClemens Marta Kwiatkowska @LeoVaradkar @antonosika @DAcemogluMIT @Yoshua_Bengio
It’s never been more clear: AI is real, and Europe needs to act. We’ve had all the warning shots and wake-up calls we need. Now the question is ‘what must be done?’ This strategy is our answer.
transformative-ai.eu
Adam Edgerton retweeted
great to see Anthropic committing to embedded evaluators.
no time to waste—let’s make this the industry standard. start voluntary today, executive action tomorrow, try to codify next Congress. scale up evaluators as we go.
from earlier this week: writing.antonleicht.me/p/sen…
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
Adam Edgerton retweeted
This is a big deal. Dario Amodei, Sam Altman, and Elon Musk are converging on something we’ve been pushing at Fathom: AI labs shouldn’t be the only ones judging whether their systems are safe.
Independent Verification Organizations, or IVOs, can provide that outside check.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
Update: we’ll be winding down the RFP a week from today!
We’ll share more about what we did, what we learned, and what comes next in the next few weeks.
You can still submit a pitch until then, but we’ll have a high bar for accepting new pitches because we want to pivot quickly to our new focus area.
REQUEST FOR PROPOSALS
What do we need to build to prepare the world for advanced AI?
>$25 billion is about to flow into AI resilience and AI-for-science from the OpenAI Foundation, the Chan Zuckerberg Initiative, and other major funders.
But there's no shovel-ready list of the essential projects to build, and no critical mass of builders ready to execute.
We're trying to fix that with The Launch Sequence, a collection of concrete projects to accelerate science, strengthen security, and adapt institutions to future advanced AI.
We're opening up The Launch Sequence for new pitches. We'll help you develop your idea, connect you with funders, and get it built. $10,000 honorarium for published proposals.
🔗 Read the RFP and submit a short pitch (200–400 words): ifp.org/rfp-launch
We’re looking for short pitches in three areas:
1. Accelerating science: What do researchers need to make breakthroughs with AI that markets and conventional grants won’t fund fast enough?
2. Strengthening security: What tools, technologies, or orgs can help ensure rapid AI advances don't undermine national security and public safety?
3. Adapting institutions: How do our institutions need to evolve to help society adapt to AI-driven change while preserving human agency?
Our advisory panel:
– @woj_zaremba, OpenAI co-founder
– @tkalil2050, CEO of Renaissance Philanthropy
– @matthewclifford, Co-founder & Chair of Entrepreneurs First, founding Chair of ARIA
On top of the $10,000 honorarium, we offer a $1,000 bounty for pitching an idea we hadn’t heard before (if we publish it), and for every successful referral.
Submissions are rolling, but we'll prioritize early pitches (within the next few weeks) and start reviewing immediately.
Adam Edgerton retweeted
When the next AI incident inevitably happens, the government will want to know what's going on inside the labs.
Let's send in independent evaluators now. Formalise their role, give them access, and build their capacity before the scramble really starts.
writing.antonleicht.me/send-…
Adam Edgerton retweeted
Coefficient Giving just issued a call for founders starting/scaling ambitious AI safety initiatives. Grants ranging from ~$200K-$200M. I especially love that they've published a list of specific initiatives they would like to see people working on. Please share with your most talented friends looking for a big, consequential problem to work on next.
Screenshots of some of the initiatives they'd like to see below, more detail here: coefficientgiving.org/tailwi….
Awesome work @EmilyOehlsen @albrgr and team.
Today we’re launching Project Tailwind, a call for founders to start ambitious new AI safety initiatives: coefficientgiving.org/tailwi…
We’re looking for great people to engage seriously with the risks of transformative AI, and to create the research, technologies, and institutions that will help humanity navigate them.
There are critical, basic problems that no one owns. Over the last several months, my team at @coeff_giving brainstormed ideas we’d be excited to fund if we could find a promising founder, and the list quickly grew to more than 200 entries. We’ve narrowed it down to our favorites, but we also expect the best founders to bring their own ideas:
coefficientgiving.org/tailwi…
We’re providing funding at three levels:
1️⃣ Pre-seed: $200k to $2m to develop an idea and build a team
2️⃣ Seed: $2m to $20m to launch and scale
3️⃣ Scale: $20m to $200m+ for proven teams to scale, or world-class teams to start
If you’re excited about something on our list, or you have another proposal for driving the field forward, you should get involved: coefficientgiving.org/tailwi…
There are more good projects than there are people to work on them. Please help us make that stop being true!
Also hi, I’m Emily! This is my first tweet. I lead AI and biosecurity grantmaking at @coeff_giving.
Adam Edgerton retweeted
Hello folks,
I’m helping run a new AI and governance conference in D.C. this November, if it sounds up your alley, please apply!
Adam Edgerton retweeted
Great news! Governor Newsom has signed SB 813 into law, a Fathom-sponsored bill that establishes the first comprehensive framework for independent verification organizations (IVOs). 🧵
Adam Edgerton retweeted
Tomorrow after briefing, stick around for: FRONTIER After Hours, this Tuesday 5:30-7:30pm at Smoke & Mirrors.
Drinks and conversation with policymakers, Hill staff, technologists, and the rest of the AI policy crowd.
RSVP: luma.com/zgs82ixi
Join us Sept. 1 for a Hill briefing on the FRONTIER Act!
@ghadfield, @sgandhi0, and @CharlieBull0ck join @Fathom_org's Jake Swanton to unpack evaluations, risk management, transparency, and oversight.
Rayburn 2043, 12-1pm ET.
RSVP ⬇️

















































