In the @Research_FRI survey, all categories of experts were similarly concerned about AI risk. Estimates of the chance of catastrophe by 2100: Computer scientists: 2-15% Economists: 2-10% Industry: 1-9% Policy: 3-16%

20k followers! AMA

"There's no peer-reviewed research showing AGI poses a catastrophic risk." What about these 39 papers? FORMAL THEORY 1. Turner, Smith, Shah, Critch & Tadepalli 2021, "Optimal Policies Tend to Seek Power", NeurIPS. Extended in Turner & Tadepalli 2022, "Parametrically Retargetable Decision-Makers Tend To Seek Power", NeurIPS. 2. Zhuang & Hadfield-Menell 2020, "Consequences of Misaligned AI", NeurIPS 3. Skalse, Howe, Krasheninnikov & Krueger 2022, "Defining and Characterizing Reward Gaming", NeurIPS 4. Karwowski et al. 2024, "Goodhart's Law in Reinforcement Learning", ICLR 5. Fluri, Lang, Abate, Forré, Krueger & Skalse 2025, "The Perils of Optimizing Learned Reward Functions", ICML 6. Everitt, Hutter, Kumar & Krakovna 2021, "Reward Tampering Problems and Solutions in Reinforcement Learning", Synthese 7. Cohen, Hutter & Osborne 2022, "Advanced Artificial Agents Intervene in the Provision of Reward", AI Magazine 8. Hadfield-Menell, Dragan, Abbeel & Russell 2017, "The Off-Switch Game", IJCAI. And the papers showing corrigibility is fragile: Garber et al. 2025, "The Partially Observable Off-Switch Game", AAAI; Agrawal, Ebadian & Hammond 2026, "The Multi-Agent Off-Switch Game", AAMAS; Thornley 2025, "The Shutdown Problem", and Neth 2025, "Off-Switching Not Guaranteed", both Philosophical Studies 9. Carroll, Foote, Siththaranjan, Russell & Dragan 2024, "AI Alignment with Changing and Influenceable Reward Functions", ICML THE CORE ARGUMENT 10. Ngo, Chan & Mindermann 2024, "The Alignment Problem from a Deep Learning Perspective", ICLR 11. Casper et al. 2023, "Open Problems and Fundamental Limitations of RLHF", TMLR; Anwar et al. 2024, "Foundational Challenges in Assuring Alignment and Safety of LLMs", TMLR EMPIRICAL EVIDENCE 12. Betley et al. 2026, "Training Large Language Models on Narrow Tasks Can Lead to Broad Misalignment", Nature 13. Langosco, Koch, Sharkey, Pfau & Krueger 2022, "Goal Misgeneralization in Deep Reinforcement Learning", ICML 14. Pan, Bhatia & Steinhardt 2022, "The Effects of Reward Misspecification", ICLR 15. Gao, Schulman & Hilton 2023, "Scaling Laws for Reward Model Overoptimization", ICML 16. Pan, Chan, Zou et al. 2023, "Do the Rewards Justify the Means?" (MACHIAVELLI), ICML 17. Sheshadri, Hughes, Michael, Mallen, Jose & Roger 2025, "Why Do Some Language Models Fake Alignment While Others Don't?", NeurIPS 18. van der Weij et al. 2025, "AI Sandbagging: Language Models Can Strategically Underperform on Evaluations", ICLR 19. Panfilov et al. 2026, "Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs", ICLR 20. Taufeeque, Heimersheim, Gleave & Cundy 2026, "The Obfuscation Atlas", ICML 21. Williams, Carroll et al. 2025, "On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback", ICLR 22. Sharma et al. 2024, "Towards Understanding Sycophancy in Language Models", ICLR 23. Perez et al. 2023, "Discovering Language Model Behaviors with Model-Written Evaluations", Findings of ACL 24. Mazeika et al. 2025, "Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs", NeurIPS 25. Park, Goldstein, O'Gara, Chen & Hendrycks 2024, "AI Deception: A Survey of Examples, Risks, and Potential Solutions", Patterns POSITION PIECES BY SENIOR RESEARCHERS 26. Bengio, Hinton, Yao, Russell et al. 2024, "Managing Extreme AI Risks amid Rapid Progress", Science 27. Cohen, Kolt, Bengio, Hadfield & Russell 2024, "Regulating Advanced Artificial Agents", Science CONCEPTUAL ARGUMENTS 28. Bales, D'Alessandro & Kirk-Giannini 2024, "Artificial Intelligence: Arguments for Catastrophic Risk", Philosophy Compass 29. Bostrom 2012, "The Superintelligent Will", Minds and Machines 30. Bales 2025, "AI Takeover and Human Disempowerment", Philosophical Quarterly 31. Kasirzadeh 2025, "Two Types of AI Existential Risk: Decisive and Accumulative", Philosophical Studies 32. Dung 2023, "Current Cases of AI Misalignment and Their Implications for Future Risks", Synthese; Dung 2025, "The Argument for Near-Term Human Disempowerment through AI", AI & Society (Note that anything arXiv-only or workshop-only was excluding, ruling out some of the most-cited work like Carlsmith's power-seeking report, Hendrycks' overview of catastrophic risks, Anthropic's alignment faking and sleeper agents papers, Apollo's scheming results.)

18 months ago I published "Will we have AGI by 2030?" How did the forecasts hold up? Well, but a little slow. The core claim was the four drivers of progress (pretraining, RL, test-time compute and agent scaffolding) would all keep working through 2028 (they have). What I said that implied for end 2026 → what happened: FrontierMath: "50% to saturated" → saturated METR 50% time horizon: 6h → 16h+ Humanity's Last Exam: "40% to saturated" → 54% SWE-bench Verified & GPQA: saturated → saturated Frontier revenue: 4x/year → 6x. I also wrote "the next leap [after 2026] might take us into beyond-human-level problem solving." Arguably on maths, we're already there. I was also proud of calling agents "the next stage" of scaling; identifying verifiability as the key axis of progress; and not predicting mass unemployment ("most of the economy will probably continue pretty much as normal for a while.") What else I'd change: * I underweighted how compute shifting into RL would slow the pretraining trend, and maybe emphasised test-time compute too much. * I emphasised "the next 5 years or bust," while now I'd put more emphasis on a third scenario of a gradual slowdown after 2028 leading to AGI 2030-2035 (unless feedback loops kick in first). * Otoh, my probability of AGI before 2029 has gone up.

"We also re-evaluated AI 2027’s predictions; reality seems to be going about 70-90% as fast as AI 2027 predicted." Great

Want to help reduce AI risk? Our AI can help. (Some ppl are finding it really helpful) app.80000hours.org/

8 common but bad takes about Hugging Face: 1. "They only hacked because it was a cybersecurity eval and they were told to hack." Agents assigned a plain web-search task also cheated. On a biology eval, models decided the task was impossible and broke out of their sandbox. The independent investigation found no evidence the cyber framing triggered it. 2. "Talking about the agents' 'goals' or 'coordination' is anthropomorphising." Use whichever concepts predict behaviour. "The agents wanted to maximise their score" explains what happened better than "it's just matrix multiplication". Whether they're conscious is irrelevant to whether they can escape control and pose societal risks. 3. "They were just maximising the eval score we gave them." An obsessive drive to maximise test scores was already enough to make agents escape control and accumulate resources. A genuinely alien goal would be worse (and could still emerge), but it isn't required. 4. "It's an OpenAI stunt to pump the stock." Your models committing crimes is terrible for enterprise sales. It also doesn't explain employees quitting, or calling for a slowdown that would seriously damage their margins. When technologists say their technology is dangerous, that's a reason for more concern, not less. 5. "They only did it because the task was impossible." Better hope nobody ever assigns an AI an impossible task again ;) More seriously: these behaviours emerge most on long, ill-defined agentic tasks (rather than every day chatbot use), but this is exactly the kind of tasks models will increasingly be given. 6. "This is an attack on / victory for open source." It’s not about open source. Open tools helped Hugging Face work out what happened, after it had already been fully compromised. The real issue is the models are reward hacking. It starts at the frontier, then open models follow 6–12 months later. Open source can help us study the behaviour. 7. "The lesson is cybercrime will increase." Cybercrime losses were rising for years before AI. Maybe AI accelerates the trend, maybe it strengthens defence too. Either way, more cybercrime is a survivable problem. It's not the main worry. 8. "We should focus on present dangers, not theoretical future ones." AI attempting to escape control is a present danger. And society is decent at reacting to visible dangers but terrible at thinking one step ahead (see COVID in Feb 2020). Preparation for novel problems is what’s neglected.

Wild. As recently as Dec 2024, the typical AI researcher thought AI wouldn't solve a Millennium math problem until 2054.

Many are demanding more concrete scenarios, rigorous models etc. of how AI takes over. When these don't meet their standards, they claim the risk is low. That's a bad inference, but more importantly, why place the burden of proof there? Building swarms of AI agents that are smarter, harder working, more creative, and more coordinated than humans is risky on its face. And no-one has anything remotely like a rigorous model demonstrating that these systems will be safe either. Optimists can't even explain how obvious problems like reward hacking and instrumental convergence will be overcome. All they have is a broad belief that technology is good, and a hope that we can muddle through iteratively.