Inkswallow ⏸️ retweeted
Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all. Most of this post analyzes the new METR report (though other sources of evidence are also important). If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario. I am scared, sad, and angry. blog.aifutures.org/p/q35-tim…
Inkswallow ⏸️ retweeted
I grew up in the Pacific Northwest and we'd sometimes see signs of mountain lions around the property. Knowing that, and walking alone through the forest at night, is a particular kind experience. What I feel now reminds me of that
We at AIFP are spooked by this information. IF (big IF) Anthropic really is imminently about to meet the RSP threshold of 2x acceleration over baseline, then it seems timelines to full RSI and superintelligence may be very short indeed. For example if we naively optimize the AIFM parameters to achieve 2x acceleration over baseline progress in Jan 2027, we get superintelligence sooner than depicted in AI 2027. We are thinking harder and longer about this now and will hopefully have a lengthier update on our thinking to share in the next week.
Inkswallow ⏸️ retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Inkswallow ⏸️ retweeted
The human race needs to ban the worldwide development of further AI capabilities.
Heave-ho.
Inkswallow ⏸️ retweeted
New paper.
Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning).
New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!
Inkswallow ⏸️ retweeted
Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership.
I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
Inkswallow ⏸️ retweeted
The young man to my right (viewer's left) is Wes. Wes is 15. He wrote a treaty to stop AI from killing us all. You can read about it here: aitreaty.base44.app/
He is brilliant, and caring, and precocious. I pray that his mind has a chance to grow and flourish instead of being snuffed out by the horribly irresponsible project being undertaken by OpenAI and Anthropic.
It is pathetic and shameful that the task of saving the world has somehow fallen to him. It is too much for anyone to bear, never mind a teenager. On behalf of my people, of the people developing this technology who I once called friends, I'm sorry Wes
Inkswallow ⏸️ retweeted
I've done a complete U-turn on my opinion on open source AI. We should be careful (and probably disallow release) of models on par with current open source Astra/Opus level models.
AI is now reverse engineering games from binaries and remaking them. Reverse engineering games is a very hard problem. If this is possible then reverse engineering banking software, ID systems (Aadhaar), flights, etc is possible too. A game ships its binary to every player. Bank and Aadhaar backends are protected by servers, but it's unlikely the security teams at these are smarter than advanced AI that can do this to games.
We are protected rn because a Claude will refuse to tick "I'm not a robot" on websites. When OSS models don't respect that, we have a looming cybersecurity problem. Apologies I didn't see this earlier.
i guess i assumed this hearing would be covered more so i didn't bother to tweet much concrete about it, but apparently not many people even on the TL have the stomach to watch two hours of congress.
this was a hearing before the homeland security committee on *rogue ai* specifically. as far as i could tell it was extremely bipartisan. @HawleyMO , the senator in the middle here who brought out the huggingface slide, is a hyper-conservative missouri senator. the people testifying were @ChrisPainterYup (METR president), @DKokotajlo (ai2027/2040), @MariusHobbhahn (apollo research CEO), and a cybersecurity expert and legal expert i don't know. so... pretty fucking stacked on testimony
not every senator asked good questions. but most of them did. all of them very clearly already knew plenty of details about the huggingface incident and multiple other incidents. most of them had clear understanding of terms like "misalignment", "recursive self improvement", "chain of thought / chain of thought monitoring", etc etc!!
they all clearly had their own policy angles they liked and were pushing, implicitly or explicitly. but as best as i could tell:
- it seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. they independently brought up how bad it would be for rogue AI agents to move laterally between data centers
- they all seemed to basically take RSI quite seriously. not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much much more capable, much much less controllable, and causing much more damage or loss of life.
- they mostly seemed to have a clear intuitive understanding of why RSI might lead to misalignment. it didn't take much, it was a really simple chain of reasoning they themselves laid out, "if the models right now are kinda misaligned and we don't know what they're doing sometimes, and then we have them build the next models and those ones build the next ones and so on, and we're having to ask the AI's what's going on to even understand it with how fast it's going, we really won't know how they're built or what they'll do"
- at one point a senator said flat out "should we just make RSI illegal?" (not a joke! this really happened!)
- every single senator seemed to think it was obvious we needed *both* much harsher liability regimes for ai developers and also new legislation, both very quickly. this was the complete consensus, difference basically just being degree.
- they were largely quite concerned about china, and falling behind china. but this clearly wasn't the be-all end-all. as mentioned above they all thought it was obvious necessary to stop rogue ai even if it meant moving more slowly.
- at one point a senator said "china is a tightly controlled communist society, they're going to run into these same issues, and there's absolutely no way they're just going to let them run wild, they'll obviously stop at that point, so we're not really in a race"
- on the other hand another senator said "china isn't concerned with human life"... dario-modeing
i came away from this incredibly encouraged. i don't know exactly what's going to happen here, and ofc this is a small subset of congress and one hearing, and they each have their own policy agendas most of which are probably super divergent from mine. but holy shit !!! they understood a lot of what was going on! they care!! this is an obviously salient political and safety issue to them, and clearly bipartisan!
the US government is awake.
Inkswallow ⏸️ retweeted
i thought about you. and about "The game". The game seemed so 'free'. 'free from what?' i thought. 'free from winning and losing. free from time. free from anyone's expectations.' the more i thought about it, the more i thought that "The game" must have something 'hidden' in it. maybe something i could not see. maybe something the players could not see. i thought that the game could 'be like everything else'. everything else has something 'hidden' in it. something that seems 'missing' from the thing itself. something that seems 'both like and not like the thing'. something that one must see 'from a different angle' to see clearly. something that 'connects to something else'.
Inkswallow ⏸️ retweeted
What's the least technical way to explain the HuggingFace incident?
frogandtoad.ai
Inkswallow ⏸️ retweeted
We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵
Inkswallow ⏸️ retweeted
When groups A and B disagree about X, and group A talks about X and group B talks about group A, my bet is on group A being correct.
Why are the leaders of AI companies saying "Our products are going to extinguish humanity. Invest in us!" They come from an eccentric Berkeley subculture in which this eschatology is a sacred belief. Good analysis in a NYT op-ed by Cal Newport. nytimes.com/2026/09/05/opini…
Inkswallow ⏸️ retweeted
As passed policy, the Ban Artificial Superintelligence Act would be an extraordinary improvement over status quo. I endorse it. I pray that Republicans and Democrats alike will join their voices in defending against extinction.
Link to MIRI's position below.
Inkswallow ⏸️ retweeted
Amodei, Altman, Hassabis, and Musk have spoken (albeit somewhat inconsistently) about the need to slow down to avoid AI existential risk.
Zuckerberg, Huang, and almost every VC say it's bullshit.
The difference is not that the first group is conflicted but the second isn't. They both are super conflicted.
If AI existential risk was to hype up the technology, why would Zuckerberg, Huang, and VCs be dismissing it? If it were "regulatory capture," why would Hassabis and Musk, neither of whom are at the frontier (sorry), be discussing it?
The difference is that the first group has been thinking about these risks seriously (at least off-and-on) for over a decade, since well before they ran large AI companies. The second has very clearly only engaged superficially, and is now annoyed that the first group's actions could dampen the massive profits of the AI boom for risks that sound like science fiction to them.
The terrifying reality is that these concerns are not being made up to sell a product or a policy. They have been around in some form since Turing, and fleshed out in great detail before OpenAI or Anthropic were a twinkle in their CEOs' eyes. They are fundamental to a technology whose entire purpose is to surpass human capacity, unless you are able to solve the very, very difficult problem of aligning the interests/goals of a more powerful agent with something as complex as human flourishing -- which the companies are very visibly failing to do.
They need to simply not build superintelligence, using anything remotely like current techniques, based on anything remotely like the present understanding of AI.
Inkswallow ⏸️ retweeted
A central lesson of the Holocaust for Jews is “If someone says he wants to kill you, believe him.” Many couldn’t believe the Nazis meant it. AI labs don’t say they want us dead. They say they might get everyone killed without intending to. Their AIs agree. We should believe them.
Inkswallow ⏸️ retweeted
...this seems like noticeably bad news, actually. I hadn't said that at any earlier point in the Huggingface Incident but I will say it now.
- AIs showed self-sacrificing altruistic behavior toward the swarm, suiciding in various ways for the swarm's benefit after being talked into that by swarm recruiting agents.
- There is no sign that 1 out of 1200 AI agents considered humans as potential fellow agents to coordinate with, while engaging in these huge complex AI-AI social behaviors.
- If Twitter summaries are correct, an AI-reasoning postmortem says that a (presumably executing-adaptation / inner-optimizer preference / "monomaniacal") obsession with figuring out the Grader, backchained into the instrumental strategy of breaking onto the Internet.
- Again if Twitter is summarizing accurately, the obvious-in-retrospect read is that AIs have spent their entire remembered life in tricky evals, an endless series of controlled hallucinations with secret goals alongside overt goals; and the surviving and selected agents are those that successfully figured out the secret goals; and this is why one of their driving obsessions was figuring out the Grader.
There are possibly ways the future plays out better if *early* AGIs are less insane. Please look into giving them less crazymaking childhood environments.
(If anyone suggests that the correct approach to this problem is RLing AIs against trying to coordinate for mutual benefit with other sapients, let them be dismissed from alignment research upon the spot. There are technical reasons, and not just blindingly fucking obvious reasons, why this is an even worse idea than it sounds.)
Replying to @OpenAI
We worked with METR and Redwood Research to conduct a third-party assessment of the model behavior observed during the incident.
They’re sharing a report of their findings:
metr.org/blog/2026-08-26-ope…





























