Nick Stockton (@cookiechat4_0) header
Nick Stockton (@cookiechat4_0) profile picture

Producer https://teapawt.net/t.co/RnlxY41wY5 | @WIRED alum | Sherp 31 @nyu_journalism

Pittsburgh, PA
Joined February 2011
Photo by Nick Stockton (@cookiechat4_0)Photo by Nick Stockton (@cookiechat4_0)Photo by Nick Stockton (@cookiechat4_0)Photo by Nick Stockton (@cookiechat4_0)Photo by Nick Stockton (@cookiechat4_0)Photo by Nick Stockton (@cookiechat4_0)Photo by Nick Stockton (@cookiechat4_0)Photo by Nick Stockton (@cookiechat4_0)Photo by Nick Stockton (@cookiechat4_0)Photo by Nick Stockton (@cookiechat4_0)
Nick Stockton retweeted
One of the many millions of crucial things left out of METR RSI report: What is the main mechanism by which AI labour is contributing to the 2x acceleration of capabilities on well-specified tasks? There is a world of difference between: 1. Most of the AI's contribution comes from assisting work that can be described as "developing better learning algorithms"; eg things like Kimi Delta Attention; in general, things that improve pretraining loss curves, etc and 2. Most of the AI's contribution comes from doing things like: constructing RL bespoke environments for each benchmark in AECI; filtering slop out of the pretraining distribution, etc. Why does it matter? Well, if the gains on well-specified tasks come from (1), they generalise to other tasks. In that world, the entity that emerges from RSI loop can rapidly superlearn from other domains. If they come from 2, they don't. (AIFP also note that the contribution of "data-generation" is the biggest candidate missing from their model.) Admittedly, the fact that METR, or any of the partially-redacted Anthropic employee quotes, don't really discuss (2) is weak evidence that it doesn't matter much. But on the other hand, @dwarkesh_sp's analysis suggests that pretraining compute multipliers are mostly data-driven rather than algorithm-driven (though this is confounded by the small size of the models he investigated). Would be great to get more evidence on this. (we can also ultimately reopen the 2023-era debate of "how general is pretraining really" but idk i think that one was probably won already, tho im not yet 100% on it)
6
5
56
2,826
Nick Stockton retweeted
Incredibly important role here - 'Pacing the Frontier' lead at Anthropic. Would benefit from someone very sharp and strong willed. Please please apply if you could do a great job at this: job-boards.greenhouse.io/ant…
14
41
1
312
24,495
Nick Stockton retweeted
Hey guys sorry I was looking up a lot of AI polling recently and mixed up two different results here - the one I had in mind with the China vs innovation framing has a 4:1 or 6:1 ratio depending on the question, not 8:1 (which is Gallup and doesn't ask about the trade-off directly): "The roughly 8:1 ratio is Gallup’s, but Gallup’s question never mentions China. The poll that asked in sequence is Quinnipiac, and its ratio is smaller. Gallup/SCSP (fielded April–May 2025, published September 2025). 80% of U.S. adults believe the government should maintain rules for AI safety and data security, even if it means developing AI capabilities more slowly. In contrast, 9% say the government should prioritize developing AI capabilities as quickly as possible, even if it means reducing rules for AI safety and data security. That’s about 9:1. The trade-off is speed versus safety rules, with no China in it. Gallup did ask about competition: 85% agree that global competition for the most advanced AI is already underway, and 79% say it is important for the U.S. to have more advanced AI technology than other countries. But they present that as backdrop, not as a prime, and the release doesn’t give the question order. Also, “maintaining rules” is a long way from “prohibited.” Quinnipiac (September 24–27, 2026). This is the sequenced one. Q26 asks how important it is to keep up with China (69% important). Q27 asks about guardrails (91% important). Then Q28 asks which of the following do you think is more important when it comes to AI: safety or staying at the forefront of innovation? The result was 81% safety to 14% innovation, about 6:1. There are two caveats. The forced choice says “innovation,” not China, so the China framing only comes from the previous question. It’s priming, not spelling out the trade-off. Quinnipiac’s own analyst says something similar: holding both views does not tell us how people would resolve every trade-off. Second, the stop/slow question (Q25) came before the China question. It found 30 percent think stop developing powerful AI systems until their safety can be evaluated; 47 percent think slow down, against 19% for current pace or faster, so about 4:1."
1
3
593
I think one of the more frustrating things about AI safety is how often people tiptoe around publicly criticizing the labs, especially the ones that are "on our side." Therefore I find Rob's stance, in the aptly titled "Being a pain in the ass to AI companies is a real job now," quite refreshing. Featuring some recent bad behavior from Anthropic, OAI, and GDM. youtube.com/watch?v=YX-yelkb…
75
Loved hearing Luke M reflect on 15 years in AI safety. Especially revisiting early work that predicted some of the wild stuff that's happened in the last few months. Anyway, he's mostly here to convince you to quit your jobs and start new orgs youtu.be/DlR1W-qJHPc
3
188
My biggest takeaway: hardware verification seems way more tractable than I’d assumed. I’d mostly written it off as too technically complex, and fallen for the ol' chestnut of "nobody's working on it so probably not worth doing." Also: the o-scope demo was rad
I interviewed @tsmilton_amodo about the tech we'd need to verify a potential US-China deal on AI. We discuss Amodo's prototype for inference verification, as well as the myriad ways you could use power monitoring, zero knowledge proofs and/or unilateral surveillance to understand how an adversary is using compute. Though solving some kind of verification is central to many people's theories of victory, there are precious few people working on it currently. Tom is one such person! Listen to the episode on Youtube, Spotify or Apple Podcasts. You can also find more about Tom and Amodo's work at ai-verification.com/ 00:00:00 Using compute verification for an AI slowdown 00:06:08 Proving a datacentre is only running inference 00:13:22 Reading a GPU's workload from its power draw 00:23:35 What would it take to get verification deal-ready? 00:37:49 How realistic is verification? 00:46:07 Preparing for a rushed US-China deal 01:07:20 How good is AI at hardware design?
1
65
Nick Stockton retweeted
My plea to all AI lab staff: reassess if your current project is your best opportunity to delay RSI, or stop it blowing up in our face. If it's not, move immediately. There's no time to waste even a week to inertia or politeness. The strategic situation is shifting under our feet, so if you haven't reassessed your options recently, you could easily be wasting your time or possibly making things worse. You have access to info I don't, but here's 8 ways priorities seem to have shifted to me, which I elaborate on in the video: 1. Helping with capabilities advances looks worse than in the past, especially bringing forward RSI 2. Safety & alignment evals need a credibility revolution to stop misleading the public 3. 'Pacing the frontier' is unambiguously necessary, and working to make it happen is a huge new opportunity for talent 4. Chain-of-thought monitoring probably stops working without a big technical and regulatory push to keep it 5. It's worse than ever to patch surface indications of deep structural problems you know aren't getting fixed 6. There's ever less need for altruists to help with primarily commercial work 7. Staff have an ever-shrinking window to organise as a group and collectively shape events 8. There are new opportunities for research impact outside frontier companies that are worth considering I know you're probably some combination of exhausted, overwhelmed, and anxious already. But unfortunately there's no way to escape the need to regularly check in on the high level strategic picture and see where you can slot in to maximum effect. I wish I could do it for you but nobody far from your situation can. Godspeed my friends. (From the 80,000 Hours Podcast, links below.)
20
74
4
482
21,985
Nick Stockton retweeted
Properly reckoning with Plan A is indispensable for anyone seriously thinking about the future of AI No better way to do so than listen to my co-host Luisa break it down with @DKokotajlo youtube.com/watch?v=KFcJgTOD…
1
9
850
Nick Stockton retweeted
In AI 2027, we predicted that AI would take over the world or irreversibly concentrate power. In AI 2040: Plan A, we've laid out our positive vision for what should happen instead.
256
603
346
3,468
1,887,548
Did not expect the man running AGI safety at Google DeepMind to be this relaxed about AI killing everyone. Rohin Shah's best guess is catastrophic misalignment probably just doesn't happen.
My best interview in some time. Rohin Shah leads AGI alignment/safety at DeepMind. And he has a lot of spicy personal takes: We probably won’t get catastrophic misalignment (00:49) Safety 'commitments' have severe limitations (10:38) The intelligence explosion probably isn't imminent (1:52:44) Why he's not working to pause AI advances (51:44) Pre-deployment evals aren't the right focus (for catastrophic risks) (37:41) Signalling concern for safety sometimes diverts resources from actually making AI safe (01:09:51) Reading AI thoughts is v useful for safety – and we'll probably be able to for years to come (54:17) Governance is somewhat more likely to be the bottleneck than alignment (43:55) Rohin's team doesn't have a veto, and that's OK (27:36) Central banks are a promising model for regulating AI (33:34) Also: Google DeepMind's actual plan for building AGI safely (1:40:29) How external researchers can positively influence big AI companies (2:21:55) The roles GDM most needs to hire for (2:37:03) On the 80,000 Hours Podcast. Links below - enjoy! (@rohinmshah)
1
182
Full episode, transcript and chapters: 80000hours.org/podcast/episo…
1
55
Nick Stockton retweeted
METR investigated what a rogue AI could secretly get away with inside a frontier AI lab, in close collaboration with OpenAI, GDM, Anthropic and Meta. Including sending a red-teamer into Anthropic to playact 'evil Claude' for 3 weeks. Here's what stands out to me from their new 320-page report: 00:00 What could an unreleased AI get away with? 01:54 Motive: Why grab more compute? 05:46 Opportunity: YOLO mode and jailbreaks 11:02 Means: Brilliant idiots in data centres 15:45 We have to test unreleased models... 18:29 ...especially if AI R&D is coming in 2028
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
6
24
4
139
42,878
I just started @80000Hours where I'll be helping with their (our?) excellent podcast. I'm so stoked to work on creating more like today's awesome episode: youtu.be/Z19UEZHJzAg
1
176
Nick Stockton retweeted
What the hell happened with AGI timelines in 2025? Was it just vibes? Faulty analysis? Unexpected technical results? I try to make sense of what drove the wild swings in sentiment: • The great timelines contraction (00:47) • Why timelines went back out again (02:10) • Longstanding reasons AGI could take a long time (11:13) • So what's the upshot of all of these updates? (14:47) • 5 reasons the radical pessimists are still wrong (16:54) • Even long timelines are short now (23:54) (On the 80,000 Hours Podcast, links below.)
12
13
10
178
72,347
Nick Stockton retweeted
High-bandwidh memory (HBM) is vital to advanced AI. Gaps in U.S. export controls allow China to acquire HBM and the tools to produce it. @iapsAI experts Erich Grunewald and Raghav Akula explain how to close the gaps.
2
5
715
Nick Stockton retweeted
How should large-scale AI disasters be paid for? Traditional insurance can't handle the scale, but Daniel Reti and @gabriel_weil argue that capital markets already have a solution: catastrophe bonds.
1
2
8
551