Stop AI🛑

OpenAI, Anthropic, and >1,300 employees at all the major AI companies think the US government should prepare to “pace” automated AI R&D. The problem: “pacing” is pretty vague, and could be pretty bad. To fix this, we pulled our team together and developed 23 low-regret policy ideas to understand and manage the risks of automated AI R&D. Here they are: 1. Frontier AI companies and relevant industry bodies should publicly share information relevant to trends and risks in AI R&D automation 2. Congress should legislate transparency about automated AI R&D risk management, incident reporting, whistleblower protections, and model behavior specifications 3. Congress should resource the Center for AI Standards and Innovation (CAISI) with a budget of at least $84 million per year and empower it to directly advise senior government officials and frontier AI companies 4. The White House should set clear roles and responsibilities of US government agencies to increase specialization across AI policy 5. Intelligence agencies should improve their collection and analysis on foreign AI development and counter threats targeting US AI companies 6. CAISI should develop guidelines for managing the risks of rapid AI capability improvement 7. CAISI should co-lead an AI Verification Consortium (AIVEC) with industry to prototype and deploy verification technologies 8. AIVEC should coordinate the creation of AI hardware testbeds and make them available to government, industry, and nonprofit partners 9. AIVEC should launch philanthropically funded prize competitions for AI verification headed by CAISI 10. AIVEC should coordinate the construction of a fully verifiable data center 11. DARPA and the NSF should set up AI verification R&D programs 12. Intelligence agencies should develop and operationalize unilateral means of AI compute monitoring 13. NSA, CAISI, CISA, and ONCD should further invest in cybersecurity resilience 14. Congress, OSTP, and the CDC should invest in biosecurity resilience 15. Congress and the Bureau of Industry and Security (BIS) should strengthen controls on US and allied semiconductor manufacturing equipment 16. Congress and BIS should close gaps in AI chip controls 17. The Federal Trade Commission (FTC), Department of Justice (DOJ), BIS, CAISI, and Congress should help industry counter adversarial distillation of US AI model capabilities 18. BIS should maintain visibility into sales of US chips 19. DOW, the IC, CAISI, and relevant Federally Funded Research and Development Centers (FFRDCs) should establish consensus security guidelines for protecting model weights from theft and prototype them in a government facility 20. Congress should ensure the US has sufficient electrical capacity to sustain AI leadership 21. Congress should ensure data centers can be constructed in America 22. The US government should use its bilateral AI dialogue with China to jointly develop guidelines for managing risks from rapid AI capability growth and prepare verification measures 23. Countries with national AI institutes should collaborate on automated AI R&D risk management guidelines and technical capacity for AI verification You can read an intro to the report over at @Noahpinion’s Substack: noahpinion.blog/p/should-we-… And read the full report here, which has a bunch more detail on each recommendation: ifp.org/preparing-for-ai-res…

I resigned from Google DeepMind bc it broke its founding promise by selling AI to the military without restrictions against killer robots or mass spying. For months, I worked to stop this but watched powerful ethicists and institutions choose silence. Here's what happened. 🧵