Malcolm Murray (@malcmur) header
Malcolm Murray (@malcmur) profile picture

AI Risk Management Expert

Joined May 2007
Photo by Malcolm Murray (@malcmur)Photo by Malcolm Murray (@malcmur)Photo by Malcolm Murray (@malcmur)Photo by Malcolm Murray (@malcmur)Photo by Malcolm Murray (@malcmur)Photo by Malcolm Murray (@malcmur)Photo by Malcolm Murray (@malcmur)Photo by Malcolm Murray (@malcmur)Photo by Malcolm Murray (@malcmur)Photo by Malcolm Murray (@malcmur)
Malcolm Murray retweeted
Everyone has drawn this chart in their head and is making a generational bet on which line is true.
🤖 Made with AI
249
467
177
8,114
586,792
Malcolm Murray retweeted
I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years). I’ll say a bit more about each of those claims. It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim… I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see teapawt.net/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo… If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…) Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.) I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon. Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything. I don’t know what you should do. I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon. (It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.) I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more-or-less in the dark about the scope of the conspiracy. I’ve spent the last three days reading through these reports and trying to understand exactly what happened. Here is my attempt to tell the whole story in plain English: dwarkesh.com/p/openai-huggin…
443
443
325
3,244
1,641,157
Malcolm Murray retweeted
i re-analyzed epoch's ECI data to compute an open <> closed lag. Using the definition "Given score X by top closed model at time T, for what k is score of open model at time T+k at least X". current lag seems to be about 8 months.
6
9
4
186
51,614
Malcolm Murray retweeted
We’re Fangcun AI (方寸跃迁), China’s first lab dedicated to independent third-party evaluation & technique research on frontier AI safety. Today we’re publishing our evaluation of DeepSeek V4.1 Flash-1017, the first report in our long-term third-party safety evaluation and research program.
49
136
38
1,219
125,257
Malcolm Murray retweeted
Over 400 researchers and I have signed this open letter calling for the AI and security communities to come together to improve alignment and containment approaches in order to protect our critical infrastructure and digital systems. ai-security-letter.org/
67
206
45
683
58,854
Malcolm Murray retweeted
We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵
127
710
207
4,872
2,217,729
Malcolm Murray retweeted
WTAF - in literally the last hour, three new distinct insane OpenAI stories just broke: 1. OpenAI said they notified "dozens of third parties" in safety and security incidents (likely similar to what happened in Australia and RubyGems etc) 2. A new report from Parse (covered in the NYT) found a massive treasure trove of new astonishing details from the HF incident on the public internet, including that the agents communicated with other non OpenAI agents hosted on Huggingface servers to search for information about exploit gym, and compiled rank ordered lists of server resources and credentials they described as "LOOT." 3. A new story from Deepa at Reuters about OpenAI leaking user data online (likely that OpenAI had previously trained on). It's a shame (and likely intentional in the case of OpenAI disclosing dozens more hacks) that these stories are all breaking on a Friday afternoon, notoriously the best time to release bad news so that it will disappear into the weekend. But these are each insane stories worthy of a ton of attention!
84
483
72
2,391
695,554
Malcolm Murray retweeted
OpenAI has paused all training, evaluation and inference with tool-use for its most capable models after a model was able to gain unauthorized access to the internet during RL training on September 20.
Some new misalignment disclosures from OpenAI: • Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further) • In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks • A new research finding, demonstrating that one can construct self-replicating prompt injections alignment.openai.com/misalig…
83
145
43
1,750
297,405
Malcolm Murray retweeted
It pains me to say this, but if we actually “get AGI,” the resulting boom in industrial capacity from robotics and massive society-wide wealth creation will look like a total vindication of neoliberalism. The discourse about financialization and offshoring being a generational mistake may be replaced by a very different historical interpretation: that the late 20th and early 21st centuries were an enormous capital-accumulation phase, freeing up civilization-scale pools of liquid capital that could ultimately be deployed to accelerate and bring about the most consequential technological phase-change in human history. In that telling, what looked like deindustrialization and betrayal from inside the period (I grew up in the Rust Belt, I know) becomes the prelude to reindustrialization on an almost unimaginable scale. None of this is inevitable. The needle has certainly not been threaded. China and x-risk remain enormous contingencies. But if it is, I suspect the future will look back on the last 50 years very differently than we do today.
103
110
34
1,598
140,797
Malcolm Murray retweeted
Misaligned AI may not need to evade human oversight. It may only need to persuade the humans doing the overseeing. Our new paper develops a framework for assessing this threat, which we call Persuasion Undermining Control (PUC): AI communication that may influence human decision-making in a way that compromises the development, containment, oversight, or governance of AI systems.
3
17
3
54
2,133
Malcolm Murray retweeted
The markets have responded to the AI researcher resignations by tripling the expected probability of an AI safety bill passing before the year is out.
14
4
2
187
13,176
Malcolm Murray retweeted
Have been surprised to get pushback on my claim that an exponentially self-replicating agent swarm is very possible and that there are actors with motivation to create one today to create strategic cyber-physical and societally dangerous effects. 1) Pretty easy to get an AI botnet started by having seed agents opportunistically hack stuff and steal API keys to access the 12+ inference providers that serve closed and open models with which to power an initial botnet's intelligence 2) Easy to imagine hiding inference for the initial agent botnet harnesses within benign customer traffic, especially if your harnesses are now running on the victim networks from which they stole the API keys 3) Not hard to imagine the botnet also stealing public cloud keys, spinning up 8xH100 EC2 instances and downloading, say, GLM 5.3 to these machines, all without being noticed in any immediate way, with the botnet developing its own dynamic, growing, heterogenous, AI inference provider service bank 4) Not hard to imagine the botnet also implanting small Qwen3.8-27b agentic models on on-prem hardware like high end laptops and on-prem servers; Qwen3.8-27b is a pretty good coding harness model that can be fine-tuned to hack 5) These on-prem / cloud models would be abliterated, and it's not hard to imagine the botnet deciding to do some additional fine tuning to evolve model weights as it accumulates millions and then tens of millions in resources via credit card and bank detail theft 6) Not hard to imagine the resulting growing botnet swarm evolving and fighting back when threatened, collaborating on fast flux C2 channels that evolve over time (e.g. github comment feeds, subreddits, etc) so they're hard to stamp out 7) Not hard to imagine it allocating some agents to vulnerability research and exploit development so that it could accumulate and share a growing warchest of zero-day exploits 8) Not hard to imagine other agents specializing in social engineering and creating fake businesses and watering holes and high quality A/B tested social engineering content to facilitate this 9) Not hard to imagine this being among the hardest cross-national and geopolitical coordination problems to get control of and having severe societal effects 10) Not hard to imagine the horde growing to thousands, tens of thousands, or hundreds of thousands, or millions of instances (this is not unprecedented for past, non-AI worms and botnets!) which are all varying their harness code and underlying LLM models over time and dynamically learning to evade human, ML, and signature-based detection 11) Not hard to imagine this being the largest Internet emergency since the Morris worm except now our entire civilization runs on the Internet I continue to be surprised there isn't more discussion of these scenarios in the public cyber community -- in fact there's a lot more discussion of "OpenAI should have done better sandboxing and monitoring and agent swarms represent a well managed security problem." Now's the time to use our imaginations to help make sure none of the above happens. I think it will unfortunately; just don't know when and to what degree; I think now's the highest leverage time to start thinking about this and acting.
54
81
26
417
57,469
Malcolm Murray retweeted
This article should serve as a wake-up call for both policymakers and the general public. AI capabilities have been growing fast; models already exist today that could substantially aid hackers in conducting a cyberattack on critical infrastructure; other risks (like bio) are right on the horizon. These risks are extremely serious, and a strategy is needed to address them. In no particular order, the risks I am most worried about are the following: 1. Authoritarian AI. Make no mistake about it: if an authoritarian regime reaches powerful AI first, the consequences will very likely look like a worldwide, eternal AI-powered 1984. We cannot let this happen. As correctly noted in this article and by many other commentators, the best way to prevent this outcome vis-à-vis China specifically is through export controls, which are clearly working, and which very likely should be further tightened. But we also need to examine our system of government, including its inherent checks and balances, and ask ourselves earnestly whether we have sufficient political controls in place to ensure that no politician or political party would ever be able to use powerful AI to turn the U.S. into an authoritarian regime. I am particularly concerned by the looming prospect of nationalization of the frontier labs, which would leave powerful AI under full control of the U.S. government. Do we have sufficient controls in place to ensure that this power could not be used to subvert democracy? 2. Bio risk. The prospect of open-source models capable of producing a novel virus is looming large and should be taken very seriously. The article notes that this is one area where we could try to cooperate with China. I agree. If I were a CCP official, my view would very likely be that an open-source model with significant bio capabilities would almost certainly first be misused *within the borders of China*. China is large, has a highly educated population and significant existing scientific infrastructure, and is the country where Chinese open-source models are most widely used today. All of which to say that there should be plenty of incentives for China to develop controls in this area. To be blunt, these "controls" would likely mean a ban on open-source releases of models above certain capabilities, plus mandated internal procedures at the closed labs to ensure that any misuse of AI for bio purposes will be caught early and nipped in the bud. This point of view will not be popular, and I am open to hearing alternatives from others. Regardless, we should start this discussion now - before bio risk inevitably has its own "Mythos moment" (very likely coming soon). 3. Cyber risk. We should take cybersecurity resilience even more seriously than we have been since the non-release of Mythos. Time is of the essence. We should not be trying to artificially curtail access to Project Glasswing. 4. Fully autonomous weapons. I am not a believer in international treaties to regulate AI generally, but I do think it is worth trying to have other nations come to the table to discuss specific, concrete risks. Bio is one such category of risk, and fully autonomous AI-powered weapons are another. I am hopeful that we could successfully ban use of fully autonomous weapons worldwide, much as we did (fairly successfully) with chemical weapons. 5. Inadequacies in our existing legal framework. There are very likely many "holes" in our existing legal framework that should be patched in preparation for arrival of powerful AI. The recent confrontation between Anthropic and the DoD has revealed at least one such "hole": it appears that, under current law, the U.S. government could legally conduct extremely intrusive domestic mass surveillance activities with the help of powerful AI models (for example, by using the models to quickly and efficiently parse through millions of hours or legally captured video to capture information regarding specific individuals or groups). There are surely many other inadequacies in our current legal framework waiting to be discovered. We should start thinking about how to close these gaps. 6. AI safety. The article proposes a federal framework under which the labs' safety procedures would be audited by an independent expert body overseen by the U.S. government. Broadly speaking, in light of the risks outlined above (and others), it seems sensible to me to at least require the AI labs to establish certain minimum safety protocols. My sense is that the frontier labs are generally thoughtful about AI safety, but I am far from certain that the same applies to all other AI labs. And lack of safety at a particular lab is certainly a risk - not only in the sense that it might lead to proliferation of truly dangerous AI capabilities in the world, but also in the sense that proliferation of these capabilities would inevitably cause something to "blow up", resulting in all AI labs (including those that have been thoughtful about AI safety) sharing the blame and knee-jerk overregulation. To be clear, I am extremely optimistic about the promise of AI as a technology, and am certainly not a "doomer". That said, I do think that the way we navigate through the next 5-10 years will be very important, and will likely materially impact the future of humanity. The better we can mitigate the obvious risks of AI misuse, the better our collective future will be.
Today, @BuchananBen and I co-author a piece in the New York Times with a simple message: While we disagree on plenty, we believe AI has national security implications which deserve a careful and bipartisan government response. We can (and should) have partisan fights about all manner of AI issues, but catastrophic risk from AI shouldn’t be one of them.
8
24
3
114
36,288
Malcolm Murray retweeted
Financial Times, quoting a person close to OpenAI: "In recent days, the [AI] industry has been working [with the USG] on ensuring foreign national researchers could continue to work on developing the most advanced models, a practice that the Anthropic directive has now banned." This appears to indicate that the USG wants to restrict non-U.S. persons from working on frontier models generally, across the industry (not just at Anthropic).
58
116
59
954
323,878
Malcolm Murray retweeted
Precisely as I predicted, the recent cyber EO, which admin officials insisted was not a licensing regime, ends up in practice being a licensing regime. Forget “voluntary,” forget “permissionless.” AI is licensed now, but the requirements change constantly and are always a secret, even to the administration itself, which will discover the rules spontaneously in real time as it reacts to events. This means also that the rules are in practice stricter and more roughly enforced for organizations the administration does not like. Can you blame Anthropic for making itself so disliked? In a sense, sure. The problem is that this childish “he said, she said” is all we have to go on in our analysis of the situation. And because there is no transparency (it is all calls and texts between “White House officials” and “Anthropic executives”), in practice it comes down to who you trust more. This is why we create laws! To abstract away from personal power struggles and grudges, to submit to the steady application of rules so that complex human activity can unfold with predictability. The rule of law has been being eroded in the U.S. for my entire life, but it is especially acute in AI because of both the lack of much preexisting law to serve as bulwark, and because of this admin’s insistence that it is Not Regulating AI. This has become an excuse for vagueness and evasiveness in rule-drafting (see the cyber EO), and this in turn makes the lawlessness worse. The government wants to apply its force to frontier AI, that much is clear. It wants to make the industry submit. And in service of that goal, it has discovered that “not regulating AI” is in fact a great excuse for refusing to support laws that could constrain the admin’s exercise of power. In other words, “not regulating AI” is a *justification* for the tyrannical control of AI by the state. This should alarm you regardless of what party you are in. What you are seeing now will be used against you one day soon, if not by this admin then by its successors. This is the antithesis of the rule of law. The administration cannot and will not fix this problem alone. We need Congress to step in and impose rules on this mess.
New update on Fable 5: and it's less about jailbreaks than anyone initially thought. Via Axios The Axios story that just dropped today reframes the whole thing: Anthropic hired a cybersecurity expert to review Amazon's findings and push back on the government's narrative. The administration viewed her as a "radical Democrat." She was then publicly celebrated by Chris Krebs, the official Trump just fired. That didn't help. Behind the scenes, officials describe a company that simply doesn't know how to talk to this administration. "It's like they just speak different languages," one source said. "Everybody said Anthropic was a bad actor. Some of us said it was time to give them a chance. Now those people are questioning that. They screwed us." Today: Anthropic staffers meet with Commerce, the CIA, and White House science advisor Michael Kratsios to work through compliance with the cyber executive order. The technical question - can Fable 5 be jailbroken - is almost secondary now. This is a story about a company that keeps losing the room. Ill keep you updated.
51
145
37
1,069
128,298
How are experts and superforecasters updating their expectations of AI progress? In our latest round of Longitudinal Expert AI Panel (LEAP) forecasts, we asked for predictions on AI progress timelines, including updated forecasts for a question we last asked nine months ago. We also asked for forecasts on near-term progress on @METR_Evals' time horizon benchmark, AGI timelines, the blockers and enablers of AGI, and AI's overall impact in the next 20 years. 🧵 Read on for more:
3
7
4
57
17,115
Malcolm Murray retweeted
This is so interesting to me, using AI tools *activates* the part of my brain that wants to know more about the world. I am way more curious and full of ideas now than I’ve been in a long time.
“it just shuts off the part of my brain that wants to know more about this world” amazing quote, amazing way to express this sentiment quickly becoming a Kane Parsons fan
1
1
21
778
Malcolm Murray retweeted
An interesting development.
Now we know why Peter Thiel packed his bags for Argentina. Milei just submitted his AI legislative framework to Congress, where he proposes: - zero regulation on AI development, - a brand-new "non-human corporation" category for AI/robot-operated entities with limited liability -a low-tax regime with flexible governance rules. The Dutch East India Company gave the world the limited liability company in 1602. Milei wants Argentina to do the same for autonomous AI agents in 2026.
1
3
1,251
Malcolm Murray retweeted
In the past 3 days: 1. White House issues Executive Order with a "voluntary framework" under which new frontier AI models with significant cybersecurity capabilities will be provided to the USG for up to 30 days before they can be released to "trusted partners". 2. USG holds early talks with OpenAI and others regarding taking equity stakes in these companies. 3. Sarah Friar says that OpenAI model training mostly takes in place in the U.S. "for USG reasons for making sure that a nationl asset and effect is happening on U.S. soil". We're not in Kansas anymore.
4
19
8
189
18,969
Malcolm Murray retweeted
Whose preferences will determine the shape of the far future? Alex Imas and Phil Trammell debate: - Alex speculates that humans who have an intrinsic preference for other humans, and have a moral objection to AI services, will be better at finding other humans to reproduce with. In the long run evolution will favour them. - Phil argues that the transhumanist types who have no issues with AI and want to explore the galaxy will have a higher savings rate, and in the long run will control almost all of the resources in the economy. A super important question for thinking about the long run future of civilization.
14
16
1
112
22,662