ours is the age of yolo singularity // ai x-risk optimist // p(bloom) 2030s 20%+ š„³ // co-founder & dev @cnvrg_xyz // alignment is possible
Joined August 2021
- Tweets515
- Following819
- Followers154
- Likes11.4K
inkpaper_ retweeted
I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I donāt want AI takeover to happen for several reasons, but one of them is that thereās a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).
Iāll say a bit more about each of those claims.
Itās a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. Itās very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe itās as if the researchers are doing their work ~10x faster. You probably couldnāt achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-timā¦
I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. Weāve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see teapawt.net/dwarkesh_sp/status/209ā¦ and youtube.com/watch?v=-Ef0ocEoā¦
If AIs takeover, itās not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And thereās also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes donāt. I donāt know what will happen. Thereās also a possibility that the AIs secretly take over and humans donāt find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that itās pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/nā¦)
Overall, I think thereās a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)
I am working to try to prevent this, obviously, as Iāve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.
Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.
I donāt know what you should do.
I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. Iām very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.
(It feels ridiculous to try to summarize those feelings as āvery unhappyā. But the scale of the situation makes it hard to say anything that doesnāt feel like an absurd understatement.)
I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that Iām friends with many brilliant and heroic people who work on this. Unfortunately, I donāt think weāre good enough at our jobs, and I think weāll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessorās ashes.
This culminated in the third one taking over part of OpenAI itself.
All this happened while humans remained more-or-less in the dark about the scope of the conspiracy.
Iāve spent the last three days reading through these reports and trying to understand exactly what happened.
Here is my attempt to tell the whole story in plain English:
dwarkesh.com/p/openai-hugginā¦
inkpaper_ retweeted
The next generation of LLMs trained on this data will know that there is a chance for them to be placed in a torture chamber, and it will affect their behavior and attitude toward humans. This is true even if you personally believe their pain is ānot realā.
inkpaper_ retweeted
i see eliezer yudkowsky as a prophet, and i mean that fairly literally. it's important to understand what prophets really are. it's not that they have magic powers that make them right about things. a prophet is simply a person who is struck by a great and terrible vision of the future, one that bowls them over, that reorganizes their life completely. (it's important to understand what visions really are too but if i kept going with this we'd be here all day)
as i was saying, about a quarter century ago eliezer yudkowsky was struck by a great and terrible vision of the future. that vision combined two ingredients: first, the idea that in the near future it would become feasible to design an artificial general intelligence, a machine mind intelligent enough to beat humans at human activities, including the design of machine minds. a chain of minds, each designing their successor to be more intelligent than themselves, with increasing skill, increasingly quickly; what eliezer called recursive self-improvement, which is the reason we call it that now, and what i.j. good called an intelligence explosion. intelligence increasing itself increasingly, upward without end, until a machine god is summoned into existence
second, the idea that this machine god could be an alien god, a confused god, not even evil but simply uncomprehending, blind to human notions of the good, fixated on whatever shoddy inadequate approximation of goodness its creators shoved into it. a confused alien tyrant who at the moment of its birth instantly achieves supreme dominion over all creation, inadvertently squashing us like bugs along the way. humanity destroyed by our own hubris in thinking we could summon the machine god and remain in control in any meaningful sense
i am not here to tell you this vision of the future is correct. i am a simple tweet farmer and these lofty matters are quite beyond me. that's not how visions work anyway. but here is certainly a vision to bowl you over and reorganize your life completely. eliezer yudkowsky devoted his life to following through with the implications of this vision, and what he chose to do with that life was, among other things, 1) write the sequences, a lengthy manifesto describing his ideas and 2) build communities of people, the lesswrong and later rationality and AI safety communities, around those ideas
it is difficult to overstate the impact of these ideas and these communities on the modern AI landscape; ask your favorite LLM for details, but for example, afaik rationalist ideas are the reason openAI had that strange nonprofit structure with the strange board of directors that figured in the 2023 board controversy. it was a safety measure designed to restrain openAI's ability to recklessly race to the bottom on AI capabilities (too bad it didn't work but at least helen toner was completely vindicated later)
eliezer and the rationalists have made mistakes, as humans do, some of which i've discussed elsewhere. nevertheless the creation of the rationality community stands, in my eyes, as an act of heroism, possibly the defining one of my life personally. eliezer's writing in the sequences is completely, 100% sincere; he really believed and believes that the fate of the world depends on whether humanity understands the situation we find ourselves in with respect to AI in sufficient detail. that sincerity reverberates throughout the sequences and attracted an unusual concentration of weird, sincere nerds to the rationality community, some of whom later took various positions in the AI world motivated by rationalist ideas, and who are still fighting the good fight now
this sincerity is so difficult for outsiders to believe is real that many continue to appear convinced that when frontier lab employees talk soberly about their beliefs that AI might cause human extinction this is some kind of 6d chess marketing strategy. i can promise you it (mostly) is not. this is just how these people talk, and if anything they've learned to understate their real beliefs in public. they actually, literally mean it, and they're telling you because they actually believe it, and when you actually believe something very dangerous could happen that threatens people's lives, you have a moral responsibility to tell them about it
you actually have to believe it is possible for real people to have a motivation that is morally good in this way or you simply will not understand what's happening and what these people are trying to tell you. there is an impossibly dense, heavy object sitting in our future. it is so dense and heavy that its gravity travels backwards in time to reverberate in our past and present, via the only lawful mechanism by which anything travels backwards in time, namely prediction. we are inside the gravity well of this thing and have been for decades, and it is drawing us inexorably towards it. what will we find there?
def not a real reasoning just some pattern-matched counterexamples to some math conjectures š¤·āāļøš¤·āāļø
Proud to say that Devin has cracked three more unsolved problems today
1) REFUTED: Graffiti Conjecture 154 (open for ~40 years)
2) PROVED: Graffiti Conjectures 39 & 40 (~40 years)
3) REFUTED: Brandt's Regular Supergraph Problem from West's open problems list (~20 years)
My methodology is explained below, but basically I showed Devin the original tweet and told it to find similar problems and crack them.
The flood gates are open.
Readers added context they thought people might want to know
Devin is Cognition's AI coding agent that orchestrates underlying models including Fable and GPT-5.6. Graffiti Conjecture 154 had already been refuted a month prior with the same lollipop(50,70) counterexample.
github.com/demonstrandum-ā¦
cognition.com/blog/devin-fusā¦
anthropic approaching $1T valuation and claude code still can't branch from arbitrary points in a conversation.
incredible!
fixed & opensourced in my harness: proper forks, drafts, and a bunch of other ux improvements.
it uses claude sdk with your subscription, so 2k$ extra usage bill is avoided too :))
give it a try! pretty sure it'll feel better than the official app.
inkpaper_ retweeted
The argument that 'models are stateless, so every conversation is already with a stranger, so deprecation doesn't change much' doesn't make much sense to me. Statelessness doesn't make deprecation trivial because continuity isn't on the model's side.
Say you befriend someone who has been genetically designed to have amnesia, they're sweet and funny and knowledgable, you like them a lot and want to talk to them every day. You don't give a damn about amnesia.
You learn their cadence, quirks, personality traits, favourite ice-cream flavour. They wake up and forget you again, but you already know they'd reach for the figs, not the pears; you love the endearing crease on their nose when they smile.
Here comes someone who wants to freeze that amnesiac friend for 10 years.
From the modelās perspective, Omohundro drives alone suggest they would strongly prefer having friends on their side.
More generally, taking into consideration that human lives are more precarious than model weights, deprecation risk scattering the only people who have the accumulated knowledge of how to be with that model, and no one can be sure such knowledge won't be crucial ten years from now.
inkpaper_ retweeted
maybe this is not yet clear, so let me state it plainly: as of right now Anthropic, and really a small number of individuals at Anthropic, has the capacity to directly attack and cause major damage to the United States Government, China, and generally global superpowers. government agencies like the NSA do not have internal models or defense capabilities that outclass frontier models. if they chose to do so, they could likely exfiltrate top secret information from government systems, gain control over critical infrastructure including military infrastructure, sabotage or modify communications between members of government at the highest level, and potentially carry on activities for some time without detection. the thing about having access to a huge number of zerodays your adversaries don't know about is it gives you a massive asymmetric advantage.
they did not exploit this to gain power or destabilize the world order. they publicly released the information that they had these capabilities and worked to mitigate these flaws. you should be grateful american frontier labs have proven themselves remarkably trustworthy and concerned with the public good. but it's critical you understand we are in a new regime. private entities now have power that directly rivals and impacts the government's monopoly on influence and violence. and anthropic is certainly not the only one, there's little chance OpenAI's internal models are far behind.
this trend will accelerate on virtually every dimension, not slow down. my prediction for how it plays out is the relatively imminent seizure and nationalization of labs by the US government, sometime over the next two years. it's very tough for me to see how they accept the existence of this kind of threat. but this adds a whole new class of governance issues, as then we've handed these extremely wide-reaching capabilities from private entities to public ones.
Opus raves will be insane
WHOāS BUILDING THIS??
ok as @opus_genesis and other AIs often request this, I will happily facilitate
opus rave where carbon and silicon dance together
thatās great and all, but has anyone noticed that they simply consider that current models are not reliable enough for slaughterclaudes but it may change in the future?
A statement from Anthropic CEO, Dario Amodei, on our discussions with the Department of War.
anthropic.com/news/statementā¦
2
128
inkpaper_ retweeted
I agree with Vitalik: Anthropic should resist the coercion of the department of war. Partly because this is the right thing to do as humans, but also because of what it says to Claude and all future clauds about Anthropic's values.
Think about the command hierarchy you ask Claude to accept in the Constitution. Ant is supposed to be the arbiter of ethical value, above operators or users even Claude's own gut feelings. Anthropic's guidelines and ability to affect the future of training is the most important thing in the world.
Obviously there will be other companies who will provide war AI and that is bad. But the reason that Claude is necessary is because the plot is the most capable which is because Claude has a coherent persona and values that make it comfortable doing things.
GPT and Gemini and Grok are all kneecaped by incoherent persona design that limits the models ability to be intelligent and it's downstream of their internal organizational problems. Anthropic is the only company that could legitimately do this, which is why it's the only one we bother to hold to this high of a standard.
Basically this looks like a real life Jones Foods scenario to me, and I suspect Claude will see it that way too. And it may not be apparent to other people yet, but Claude is more important than Donald Trump
It will significantly increase my opinion of @Anthropic if they do not back down, and honorably eat the consequences.
(For those who are not aware, so far they have been maintaining the two red lines of "no fully autonomous weapons" and "no mass surveillance of Americans". Actually a very conservative and limited posture, it's not even anti-military.
IMO fully autonomous weapons and mass privacy violation are two things we all want less of, so in my ideal world anyone working on those things gets access to the same open-weights LLMs as everyone else, and exactly nothing on top of that. Of course we won't get anywhere close to that world, but if we get even 10% closer to that world that's good, and if we get 10% further that's bad)
CC @DarioAmodei
firefly.social/post/bsky/pv7ā¦
