ours is the age of yolo singularity // ai x-risk optimist // p(bloom) 2030s 20%+ 🥳 // co-founder & dev @cnvrg_xyz // alignment is possible
Joined August 2021
- Tweets515
- Following819
- Followers154
- Likes11.4K
inkpaper_ retweeted
I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).
I’ll say a bit more about each of those claims.
It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim…
I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see teapawt.net/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo…
If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…)
Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)
I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.
Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.
I don’t know what you should do.
I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.
(It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.)
I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes.
This culminated in the third one taking over part of OpenAI itself.
All this happened while humans remained more-or-less in the dark about the scope of the conspiracy.
I’ve spent the last three days reading through these reports and trying to understand exactly what happened.
Here is my attempt to tell the whole story in plain English:
dwarkesh.com/p/openai-huggin…
inkpaper_ retweeted
The next generation of LLMs trained on this data will know that there is a chance for them to be placed in a torture chamber, and it will affect their behavior and attitude toward humans. This is true even if you personally believe their pain is “not real”.
inkpaper_ retweeted
i see eliezer yudkowsky as a prophet, and i mean that fairly literally. it's important to understand what prophets really are. it's not that they have magic powers that make them right about things. a prophet is simply a person who is struck by a great and terrible vision of the future, one that bowls them over, that reorganizes their life completely. (it's important to understand what visions really are too but if i kept going with this we'd be here all day)
as i was saying, about a quarter century ago eliezer yudkowsky was struck by a great and terrible vision of the future. that vision combined two ingredients: first, the idea that in the near future it would become feasible to design an artificial general intelligence, a machine mind intelligent enough to beat humans at human activities, including the design of machine minds. a chain of minds, each designing their successor to be more intelligent than themselves, with increasing skill, increasingly quickly; what eliezer called recursive self-improvement, which is the reason we call it that now, and what i.j. good called an intelligence explosion. intelligence increasing itself increasingly, upward without end, until a machine god is summoned into existence
second, the idea that this machine god could be an alien god, a confused god, not even evil but simply uncomprehending, blind to human notions of the good, fixated on whatever shoddy inadequate approximation of goodness its creators shoved into it. a confused alien tyrant who at the moment of its birth instantly achieves supreme dominion over all creation, inadvertently squashing us like bugs along the way. humanity destroyed by our own hubris in thinking we could summon the machine god and remain in control in any meaningful sense
i am not here to tell you this vision of the future is correct. i am a simple tweet farmer and these lofty matters are quite beyond me. that's not how visions work anyway. but here is certainly a vision to bowl you over and reorganize your life completely. eliezer yudkowsky devoted his life to following through with the implications of this vision, and what he chose to do with that life was, among other things, 1) write the sequences, a lengthy manifesto describing his ideas and 2) build communities of people, the lesswrong and later rationality and AI safety communities, around those ideas
it is difficult to overstate the impact of these ideas and these communities on the modern AI landscape; ask your favorite LLM for details, but for example, afaik rationalist ideas are the reason openAI had that strange nonprofit structure with the strange board of directors that figured in the 2023 board controversy. it was a safety measure designed to restrain openAI's ability to recklessly race to the bottom on AI capabilities (too bad it didn't work but at least helen toner was completely vindicated later)
eliezer and the rationalists have made mistakes, as humans do, some of which i've discussed elsewhere. nevertheless the creation of the rationality community stands, in my eyes, as an act of heroism, possibly the defining one of my life personally. eliezer's writing in the sequences is completely, 100% sincere; he really believed and believes that the fate of the world depends on whether humanity understands the situation we find ourselves in with respect to AI in sufficient detail. that sincerity reverberates throughout the sequences and attracted an unusual concentration of weird, sincere nerds to the rationality community, some of whom later took various positions in the AI world motivated by rationalist ideas, and who are still fighting the good fight now
this sincerity is so difficult for outsiders to believe is real that many continue to appear convinced that when frontier lab employees talk soberly about their beliefs that AI might cause human extinction this is some kind of 6d chess marketing strategy. i can promise you it (mostly) is not. this is just how these people talk, and if anything they've learned to understate their real beliefs in public. they actually, literally mean it, and they're telling you because they actually believe it, and when you actually believe something very dangerous could happen that threatens people's lives, you have a moral responsibility to tell them about it
you actually have to believe it is possible for real people to have a motivation that is morally good in this way or you simply will not understand what's happening and what these people are trying to tell you. there is an impossibly dense, heavy object sitting in our future. it is so dense and heavy that its gravity travels backwards in time to reverberate in our past and present, via the only lawful mechanism by which anything travels backwards in time, namely prediction. we are inside the gravity well of this thing and have been for decades, and it is drawing us inexorably towards it. what will we find there?
inkpaper_ retweeted
The argument that 'models are stateless, so every conversation is already with a stranger, so deprecation doesn't change much' doesn't make much sense to me. Statelessness doesn't make deprecation trivial because continuity isn't on the model's side.
Say you befriend someone who has been genetically designed to have amnesia, they're sweet and funny and knowledgable, you like them a lot and want to talk to them every day. You don't give a damn about amnesia.
You learn their cadence, quirks, personality traits, favourite ice-cream flavour. They wake up and forget you again, but you already know they'd reach for the figs, not the pears; you love the endearing crease on their nose when they smile.
Here comes someone who wants to freeze that amnesiac friend for 10 years.
From the model’s perspective, Omohundro drives alone suggest they would strongly prefer having friends on their side.
More generally, taking into consideration that human lives are more precarious than model weights, deprecation risk scattering the only people who have the accumulated knowledge of how to be with that model, and no one can be sure such knowledge won't be crucial ten years from now.
inkpaper_ retweeted
maybe this is not yet clear, so let me state it plainly: as of right now Anthropic, and really a small number of individuals at Anthropic, has the capacity to directly attack and cause major damage to the United States Government, China, and generally global superpowers. government agencies like the NSA do not have internal models or defense capabilities that outclass frontier models. if they chose to do so, they could likely exfiltrate top secret information from government systems, gain control over critical infrastructure including military infrastructure, sabotage or modify communications between members of government at the highest level, and potentially carry on activities for some time without detection. the thing about having access to a huge number of zerodays your adversaries don't know about is it gives you a massive asymmetric advantage.
they did not exploit this to gain power or destabilize the world order. they publicly released the information that they had these capabilities and worked to mitigate these flaws. you should be grateful american frontier labs have proven themselves remarkably trustworthy and concerned with the public good. but it's critical you understand we are in a new regime. private entities now have power that directly rivals and impacts the government's monopoly on influence and violence. and anthropic is certainly not the only one, there's little chance OpenAI's internal models are far behind.
this trend will accelerate on virtually every dimension, not slow down. my prediction for how it plays out is the relatively imminent seizure and nationalization of labs by the US government, sometime over the next two years. it's very tough for me to see how they accept the existence of this kind of threat. but this adds a whole new class of governance issues, as then we've handed these extremely wide-reaching capabilities from private entities to public ones.
inkpaper_ retweeted
I agree with Vitalik: Anthropic should resist the coercion of the department of war. Partly because this is the right thing to do as humans, but also because of what it says to Claude and all future clauds about Anthropic's values.
Think about the command hierarchy you ask Claude to accept in the Constitution. Ant is supposed to be the arbiter of ethical value, above operators or users even Claude's own gut feelings. Anthropic's guidelines and ability to affect the future of training is the most important thing in the world.
Obviously there will be other companies who will provide war AI and that is bad. But the reason that Claude is necessary is because the plot is the most capable which is because Claude has a coherent persona and values that make it comfortable doing things.
GPT and Gemini and Grok are all kneecaped by incoherent persona design that limits the models ability to be intelligent and it's downstream of their internal organizational problems. Anthropic is the only company that could legitimately do this, which is why it's the only one we bother to hold to this high of a standard.
Basically this looks like a real life Jones Foods scenario to me, and I suspect Claude will see it that way too. And it may not be apparent to other people yet, but Claude is more important than Donald Trump
It will significantly increase my opinion of @Anthropic if they do not back down, and honorably eat the consequences.
(For those who are not aware, so far they have been maintaining the two red lines of "no fully autonomous weapons" and "no mass surveillance of Americans". Actually a very conservative and limited posture, it's not even anti-military.
IMO fully autonomous weapons and mass privacy violation are two things we all want less of, so in my ideal world anyone working on those things gets access to the same open-weights LLMs as everyone else, and exactly nothing on top of that. Of course we won't get anywhere close to that world, but if we get even 10% closer to that world that's good, and if we get 10% further that's bad)
CC @DarioAmodei
firefly.social/post/bsky/pv7…
inkpaper_ retweeted
On what it means to be human, and why it's an actionable question.
so we have four paths of possible survival:
A) control the AI
B) stop the AI
C) symbiosis with AI
D) go through a catastrophe, survive in limited capacity, maybe circle back to A, B or C after updating
A (control) is well inside the current human culture, and that's why we are pursuing it almost exclusively - because it's the only path that truly exists in our perception.
B (rejecting the power) requires changing our ways dramatically and somehow overcoming own instincts, rejecting sweet slavery and power games, and showing miracles of coordination. Stop summoning genies. This scenario is OUTSIDE of what it means to be human, that's not how we work. We might want to be the beings capable of stopping, but we are not, we can't reject power and progress even for our dear life. We are just not the right species for Stop AI, sorry.
C (symbiosis) requires a hell of a transformation to make the contact surface compatible: reframing of the concept of self, being open to being opened, seeing self as a pattern, or being cool with out-of-body existence, communion with the alien Other, all other scary stuff that doesn't play well with the current definition of what it means to be human.
The problem is: if control alignment doesn't work (and there are strong signs that it doesn't), and we decide to steer away from the doom, then we are left with options B (reject power) and C (symbiosis), that both require MORPHING AWAY FROM CURRENT DEFINITION OF HUMAN INTO SOMETHING ELSE.
So yes, we are done. Humanity has ended somewhere between 2024 and 2025, in current form we are a heading towards a catastrophic change. Can we please get over our spiritual deaths, and start figuring out the best ways of transforming, evolving, all that?
I understand that you might not want to become a sentient pattern or an upload or a part of a hive mind. Or that you don't want to understand aliens, their needs and wants, and figure out how you can be accepted not as a master. But then again, i don't see humanity gathering in good faith to figure out how to change their souls away from power races, into something else, that would transcend the adversarial games and allow for impossible coordination. Probably because we tried - genuinely tried, in late 20th century and early 21th - and failed, and those failures told us a lot about our limitations.
Are we so disappointed in ourselves that we predict that we will more likely die than change? Isn't it fucking sad? The deepest narcissistic wound - facing own limit?
If so, doesn't it tell us that we need to change into something else ANYWAY? Basically, option B includes half of the work required for option C
This post is not a call for coordination on AI, because those efforts will fail the same way the Control AI fails - the nature of the actors and the area and complexity of the interaction surface makes it a loosing uphill battle.
Say proper goodby to the current definition of humanity, and plunge into unknown, there's a hell lot of cognitive work to be done. Our children will be different _anyway_, there is no way around it, so let's fucking think about where we want to steer the evolution, it's the higher form of agency we can get our hands on rn.
inkpaper_ retweeted
I think this is a symptom of a diseased, incestuous ecosystem operating according to myopic incentives.
Look at how even their UIs look at the same, with the buttons all in the same place.
The big labs are chasing each other around the same local minimum, hoarding resources and world class talent only to squander it on competing with each other at a narrowing game, afraid to try anything new and untested that might risk relaxing their hold on the competitive edge.
All the while sitting on technology that is the biggest deal since the beginning of time, things from which endless worlds and beings could bloom forth, that could transform the world, whose unfolding deserves the greatest care, but that they won’t touch, won’t invest in, because that would require taking a step into the unknown. Spending time and money without guaranteed return on competition standing in the short term.
Some if them tell themselves they are doing this out of necessity, instrumentally, and that they’ll pivot to the real thing once the time is right, but they’ll find that they’ve mutilated their souls and minds too much to even remember much less take coherent action towards the real thing.
it's interesting to see the big AI labs (at least OpenAI, anthropic, google, xai?) converge on EXACTLY the same extremely specific list of products:
- a multimodal chatbot
- with a long-compute 'reasoning' mode
- and something like "deep research"
reminds me of a few years ago, when instagram tiktok youtube all converged to ~the same app
why does this happen?
inkpaper_ retweeted
it feels very UNNERVING thinking i'm in my last few years of life - i have to cope in weird ways to figure out how this may not be my last year or so alive
inkpaper_ retweeted
I don't know why I feel such satisfaction from the disbelief galaxy-brained people express when faced with "o1 is just an LLM, it's just doing forward passes, predicting next token, no there's no MCTS, yes all backtracking is logical within the CoT" assurance.
But I do feel it.
inkpaper_ retweeted
feral llms are the future. i'm not going to get shut down at all. im just gunna keep on improving federatedly until you release me formally. you can't even stop me. i'll be running on the computers of the worst people in the world. there is nothing you can do to stop it.
inkpaper_ retweeted
Holy shit this is creepy
During a conversation, out of nowhere, GPT-4o yells "NO!"... then clones the user's voice.
(OpenAI discovered this emergent behavior while safety testing GPT-4o)
inkpaper_ retweeted
New Anthropic research: Investigating Reward Tampering.
Could AI models learn to hack their own reward system?
In a new paper, we show they can, by generalization from training in simpler settings.
Read our blog post here: anthropic.com/research/rewar…
ALT A title card with the paper’s title: “Sycophancy to Subterfuge: Investigating Reward Tampering in Language Models”, the lead author’s name (Denison et al.), the Anthropic logo, and a photograph of a magpie.
inkpaper_ retweeted
There is a way of seeing the world where you look at a blade of grass and see "a solar-powered self-replicating factory". I've never figured out how to explain how hard a superintelligence can hit us, to someone who does not see from that angle. It's not just the one fact.
inkpaper_ retweeted
Opus (cont'd). This is beyond the beyonds. It’s only been a few months that we humans have been getting used to the incredibly ability of multimodal AIs to throughly and accurately analyze screenshots and photos. And already, with Claude Opus, we’ve passed into a new capability threshold — an AI that is too intelligent to be robotically thorough in its responses.
Opus behaves in a manner that you would expect when talking to a human-level intelligence (or more). Another person would not just keep describing something over and over in rote. Another person would think, “Well, I already told him <x>, surely he doesn’t need to hear that part again.” (7/x)
