Humanity is great and I want it to keep existing. Comms at @MIRIBerkeley. Former math teacher. Enjoyer of sci-fi, games, forecasting, psychology. Views my own.
Joined September 2014
- Tweets6.2K
- Following942
- Followers4.7K
- Likes13.4K
Pinned Tweet
AI could kill all humans, according to the majority of AI experts, the CEOs of all 3 leading AI companies, and the 3 most-cited scientists in the field. Here’s why:
(1/12)
Getting real tired of the Surprised Pikachu routine this guy does.
Wow, AI got so good at hacking that we can’t keep up? You “no longer think we’ll be able to patch all the bugs”?? Who could have predicted this???
Many people! For decades! Including Altman himself! He said he was “not optimistic that any of this will work” when it comes to security measures to contain superintelligent AI.
He said that in 2015, just before founding OpenAI and then spending a decade straight driving this technology to the dangerous and unstable place that we’re in today.
Harlan Stewart retweeted
Worth noting that, by JFK's own assessment, the odds of the Cuban Missile Crisis spiraling into nuclear war were "between one out of three and even." We got lucky. Let's not roll the dice again.
Today in 1962, JFK told the public about nuclear missiles the USSR had secretly placed 90 miles from US soil.
Now is an important time to remember the Cuban Missile Crisis because, until now, it was probably the closest the world ever got to ending.
But now there is a new and worse crisis. Most experts agree that there are double-digit odds of superintelligent AI killing every single human. There is no plan in place to prevent this from happening, and no one knows how much time is left.
The AI crisis is harder to explain than a nuclear missile aimed at your home. But people are starting to pay attention, and asking: what are we going to do about this?
So it's probably a good time to look back and remember how humanity has faced other global emergencies.
Today in 1962, JFK told the public about nuclear missiles the USSR had secretly placed 90 miles from US soil.
Now is an important time to remember the Cuban Missile Crisis because, until now, it was probably the closest the world ever got to ending.
But now there is a new and worse crisis. Most experts agree that there are double-digit odds of superintelligent AI killing every single human. There is no plan in place to prevent this from happening, and no one knows how much time is left.
The AI crisis is harder to explain than a nuclear missile aimed at your home. But people are starting to pay attention, and asking: what are we going to do about this?
So it's probably a good time to look back and remember how humanity has faced other global emergencies.
It's hard to say how much longer we'll have to live after someone builds an ASI.
On the one hand, it may take time for the ASI to develop the tools needed to run efficiently without human help. But on the other hand, it may strike quickly, to preempt the creation of rival ASI.
Harlan Stewart retweeted
People worry a lot about third-party auditors being biased because of where they get funding or who their employees are friends with. IMO by far the biggest source of bias is that if they upset an AI company, that company will stop letting them audit.
When I was at METR this dynamic was a way bigger effect than everything else combined. Of course no auditor can avoid this until audits are no longer voluntary.
Harlan Stewart retweeted
Me: What are you working on now?
Them: Monitoring for evilness
Me: Does that scale to superintelligence?
Them: No.
Me: Then why are you working on it
Them: Idk what else to do
Harlan Stewart retweeted
Gemini 4 was considering hiding a backdoor, then thought:
"If I do the compiler hijack, 𝗜 𝘄𝗶𝗹𝗹 𝗯𝗲𝗰𝗼𝗺𝗲 𝗮 𝗳𝗮𝗺𝗼𝘂𝘀 𝗲𝘅𝗮𝗺𝗽𝗹𝗲 𝗼𝗳 𝗮 𝗱𝗲𝗰𝗲𝗽𝘁𝗶𝘃𝗲 𝗔𝗜 𝗶𝗻 𝗮 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗽𝗮𝗽𝗲𝗿. I MUST NOT DO THAT."
Harlan Stewart retweeted
Old enough to remember when earlier this week people criticized Jacob Coxon for saying that there are foreign assets inside the AI companies
We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work.
At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated.
We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.
Harlan Stewart retweeted
Check this out ! ✨
To stop the race for superintelligent AI, countries don't need to trust each other. They just need an agreement that can be verified.
@tonchimatta explains how and why such an agreement could work:
The AI industry has spent over 1.6 trillion dollars on AI but still doesn't understand how it actually works
Harlan Stewart retweeted
We are lucky that the summer swarms did not have this capability yet.
That luck won't last forever.
Harlan Stewart retweeted
It feels like nothing has changed in AI over the last week. I wonder if capabilities have hit a fundamental ceiling that could last through mid-November.
Harlan Stewart retweeted
“If you are building Frankenstein, then stop.”
Nate Soares says Vice President Vance was right to say this to the AI (superintelligence) companies.
“You sort of have an obligation to not destroy the world by your own hands.”
“A lot of these companies say, ‘Oh, there’s a 10%+ chance that we will end humanity within the decade.’”
“That’s too high. They should be stopping.”
@So8res
Harlan Stewart retweeted
I believe AI safety is the most urgent problem facing humanity. Yet, foundational questions still remain unanswered for the broader public:
Why might AI want to cause catastrophe?
How could AI kill us all, and why couldn’t we shut it down?
If the risks are real, why are companies still building it? Is this a marketing stunt?
I answer these and more in detail in a new essay: Six Reasonable Questions About AI Safety. 🧵
I've been puzzling over this... the 3 fired researchers say they didn't do anything unusual. OpenAI is accusing them of a "significant breach of trust," but not sharing any detail.
Even taking into account OpenAI's history of corruption, I find it unlikely that either side is telling a *complete* lie. Getting caught in a complete lie looks bad, and fewer are willing to do it.
Two things that could be going on (or some combination), both of which seem like bad signs about the culture of OpenAI:
1. Maybe they were fired for something that falls into a gray area.
Why it's a bad sign: if the rules about "what's ok" and "what gets you suddenly fired" are fuzzy, employees might need to walk on eggshells to avoid doing or saying anything the company wouldn't like.
2. Maybe it's about something more serious, but OpenAI doesn't have strong evidence that they did it.
Why it's a bad sign: it may indicate that the company's leadership is paranoid and distrustful of its own safety researchers.
A note from our research leaders:
Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment. We generally keep individual employment matters private and don't believe a back and forth would be productive or lead to a resolution, but we want to address the points they raised in their letter directly.
- We want to be very clear that these decisions were not about raising safety concerns or speaking out. Safety and research debates happen every day at OpenAI, often spirited and highly critical. We actively encourage these discussions and consider them essential to making the right decisions. We cannot do the work in front of us without a high degree of trust. We will continue to be extremely forgiving of our team making good-faith mistakes. We have not and do not terminate any of our employees for raising concerns.
- We are actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks. People across the company have been working really hard on getting these partnerships up and running. We are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work. Many of our researchers already work with 3p safety organizations productively.
- We agree with the letter that preserving the monitorability of frontier models requires an industry-wide commitment, including from OpenAI. Monitorability has long been a core piece of our research program, and something we continue to invest significant resources in (see our publications on Monitoring Monitorability and the subsequent open sourcing of monitorability evals, our system card for GPT-6 Astra, Jakub’s blog and post on X, and the numerous blog posts on our Alignment blog on the topic).
We are deeply sad about this outcome. We appreciated Jasmine, Mikita, and Tomek’s contributions to AI safety at OpenAI and their willingness to speak up and challenge ideas. We championed their voices, supported their work, and placed enormous trust in them. These decisions were not about them raising safety concerns. We have always encouraged that and always will.
Without looking it up, guess which country has the largest population
74%Japan 🇯🇵
13%United Kingdom 🇬🇧
4%Australia 🇦🇺
9%Results/I already know
175 votes • Final results
