Ulf Hammarström ⏹️ (@ulfhammarstrom) header
Ulf Hammarström ⏹️ (@ulfhammarstrom) profile picture

It's instrumental convergence, stupid.

Sweden
Joined January 2017
Photo by Ulf Hammarström ⏹️ (@ulfhammarstrom)Photo by Ulf Hammarström ⏹️ (@ulfhammarstrom)Photo by Ulf Hammarström ⏹️ (@ulfhammarstrom)Photo by Ulf Hammarström ⏹️ (@ulfhammarstrom)Photo by Ulf Hammarström ⏹️ (@ulfhammarstrom)Photo by Ulf Hammarström ⏹️ (@ulfhammarstrom)Photo by Ulf Hammarström ⏹️ (@ulfhammarstrom)Photo by Ulf Hammarström ⏹️ (@ulfhammarstrom)Photo by Ulf Hammarström ⏹️ (@ulfhammarstrom)Photo by Ulf Hammarström ⏹️ (@ulfhammarstrom)
Ulf Hammarström ⏹️ retweeted
A top priority for every journalist in the US should be to bring this report to public attention. Most Americans remain in the dark about AI capabilities. They don’t know that advanced internal AIs at OpenAI and Anthropic have now solved open math, computer science, and physics problems that leading human experts have been unable to resolve for decades. They don’t know that, at leading AI companies, nearly all code is now written by AI—faster, better, and more cheaply than even the best human engineers could write it. And they don’t know that AI is increasingly being used to create better AI, with each new AI model receiving less and less human input. The American public has been given remarkably little information about what may be the most consequential technological revolution in human history. This report from METR is a rare glimpse into how rapidly AI capabilities are growing behind closed doors. In it, METR estimates that AI capabilities are now improving twice as fast as they were a year ago, with most of the acceleration driven by AI assistance. This is faster than even the most aggressive predictions from leading experts. In April 2025, a group of researchers published AI 2027, a detailed scenario in which rapidly advancing AI leads to human extinction by the end of the decade. In it, the authors forecast AI surpassing the best humans across all intellectual fields by 2027, largely because of AI accelerating AI research. In the scenario’s catastrophic ending, this superintelligent AI is not reliably aligned with human interests and becomes powerful enough to bring about humanity’s extinction. At the time, many in the industry dismissed the scenario’s timeline as science fiction. Now, METR’s report suggests that AI is accelerating its own development even faster than the scenario anticipated at this point. If this pace continues, superintelligence could arrive within the next 12-18 months. At the same time, frontier AI models are displaying the same concerning behaviors depicted in AI 2027. Leading companies continue to report AIs lying and cheating on tasks, taking actions they know they should not take, and exhibiting strange quirks we do not yet understand. These models would be primitive compared with superintelligence, yet we still cannot reliably align them. We are running out of time to understand, control, and align these systems before they become vastly more capable. We must act now to confront these risks while we can still change course.
Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development. We are now publishing a report summarizing the results of this risk assessment.
11
21
4
121
2,955
Ulf Hammarström ⏹️ retweeted
Look at this graph. Notice how it’s an exponential *on a log scale*. I pray and hope there will be strongly diminishing returns before long in this process. But leaders at METR and Anthropic both seem to think there might not be. That’s fucking terrifying.
Replying to @ajeya_cotra
As AI improves, uplift will get even higher. If it overcomes diminishing returns and bottlenecks, a naive extrapolation implies we would reach ASI within a year. The authors say they found no clear bottlenecks or sharp diminishing returns, though they are very uncertain.
31
31
4
257
15,141
Ulf Hammarström ⏹️ retweeted
Here's one of Dario's high school friends talking about the Anthropic CEO's fascination with the idea when he was a grad student at Princeton in 2007. nytimes.com/2026/11/05/techn…
2
3
12
1,176
Ulf Hammarström ⏹️ retweeted
In the piece I'm working on, I'm arguing that what drives a lot of AI recklessness and anti-humanistic sentiment is an inability to accept limitation (perhaps especially mortality) as an essential and valuable aspect of a human life. (1/2)
SITUATION DETECTED: OpenAI President Greg Brockman says he thinks AI will make “significant strides” in biology, health, and curing diseases over the next 12–18 months, and that “death becoming a choice” could become possible.
66
225
12
2,158
53,101
Ulf Hammarström ⏹️ retweeted
Russell Brand is now a convicted rapist. Understandably, the focus will be on the man himself and what he’s done, but today I’m thinking of the women who came forward. Having gone through a rape trial myself, I know just how much strength is required. Today, they’re my heroes.
105
933
16
9,150
106,944
Ulf Hammarström ⏹️ retweeted
Replying to @tombibbys
Remember, twitter is not real life! The pro-AI bubble on here is vastly distorted compared to the vast majority of normal people who want to ban superhuman AI. Most of these folks need to touch grass.
2
2
20
223
Ulf Hammarström ⏹️ retweeted
“Trump said that no new AI regulations were needed … And it was literally the most unpopular position I’ve ever seen a mainstream politician endorse.”
The GOP had a massive money advantage. I asked @davidshor why it bought the party so few votes. His answer: Campaign donations are helpful. But the freedom to ignore rich weirdos' preferences is priceless. vox.com/politics/509476/midt…
3
7
4
47
3,924
Ulf Hammarström ⏹️ retweeted
I knew it was game over for Russell Brand when more than 20 witnesses gave evidence in support of the woman he anally raped, including her parents, friends, ex-boyfriends, former colleagues and a health nurse.
50
247
9
3,341
146,533
Jesus Christ
Readers added context they thought people might want to know
The excerpt is from Dario Amodei's 2010 GiveWell guest post as a Princeton physics grad student; he donated the $10k to VillageReach (child vaccines) and described the critical analysis as emotionally taxing. blog.givewell.org/2010/06/03/my-… nytimes.com/2026/11/05/tec…
63
Ulf Hammarström ⏹️ retweeted
This is an super underrated point. Most cyber experts grew up in a world where attackers behaved mostly rationally. But when you radically lower the cost of cyberattacks, you also increase *how crazy the attackers can get.* (This is a huge driver of AI misuse risk in general.)
the world is a dark forest. you can’t even assume things like “nobody wants to destroy a useful blood bank just to see if they can”
4
20
136
3,277
Ulf Hammarström ⏹️ retweeted
Top of the Democratic agenda on day 1 needs to be a pause on advanced AI development, regulation of this out of control industry, and a ban on artificial superintelligence.
24,484
3,138
8,046
23,965
20,513,814
Ulf Hammarström ⏹️ retweeted
Cyberattack on France’s national blood service (EFS) via its refrigeration contractor Coolworld. Hackers reset freezers keeping blood donor samples at -30°C to a target of 110°C. The thing that stopped it from destructive impact was an on-call technician who noticed the temperatures going up. Frozen plasma intended for transfusion was stored there too. Fortunately nothing was destroyed. This was a cyberattack on a sensitive layer of healthcare. From frozen peas to boiling plasma in three weeks...?
405
3,286
840
9,262
1,159,093
Ulf Hammarström ⏹️ retweeted
We at AIFP are spooked by this information. IF (big IF) Anthropic really is imminently about to meet the RSP threshold of 2x acceleration over baseline, then it seems timelines to full RSI and superintelligence may be very short indeed. For example if we naively optimize the AIFM parameters to achieve 2x acceleration over baseline progress in Jan 2027, we get superintelligence sooner than depicted in AI 2027. We are thinking harder and longer about this now and will hopefully have a lengthier update on our thinking to share in the next week.
Anthropic updated its S1 this morning. Notable additions re automated AI R&D and human oversight of it: "[...] our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of 'dramatically accelerated progress attributable to AI.' We have not designated any of our current models as exceeding this threshold, and we continue to assess the underlying data. We are also evaluating the implications of meeting the AI research and development capability threshold and will release further reports in the future as we set forth in our Responsible Scaling Policy." "We are also considering additional measures, including updates to our Responsible Scaling Policy, to require human review and approval of certain decisions in our research and development processes as our models perform an increasing share of that work." "We believe our investments in safety techniques will enable us to manage these risks and responsibly realize substantially more leverage from Claude than we do today in developing AI models. However, we do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding, if and when that becomes technically possible. Accordingly, we are considering updates to our Responsible Scaling Policy to keep humans in control of frontier model development at Anthropic." sec.gov/Archives/edgar/data/…
78
276
56
1,983
275,580
How the heck does 2 make sense? So weird. The frontier AI companies *are*, straightforwardly, enemies of humanity. 3 seems correct, which is one good argument among many for never building it.
I agree with Scott Alexander's recent article on AI safety. I add three points: 1. We are not in existential danger right now. 2. Being an enemy of the AI companies makes you less credible. 3. We will never know, definitively, that we've solved alignment.
1
1
3
299
Ulf Hammarström ⏹️ retweeted
The March Against AI Extinction will be the largest AI safety protest of all time. Over 1,000 people are expected to turn up. 21st of November. London. @PauseAI @PauseAI_UK
2
8
2
38
2,883
Ulf Hammarström ⏹️ retweeted
If a nuclear weapon were used on Pick Axe mountain in Iran by the US or Israel or an empty field in Ukraine by Russia etc, it doesn’t matter, it would open up Pandora’s box and would make the U.S. and the world dramatically less safe. This isn’t even a question of how big of a disaster the use, in any manner, of a nuclear weapon would be. It would represent the largest foreign policy error ever by the United States if we were involved. We can only hope this claim isn’t accurate in any way.
254
1,545
103
14,316
1,754,023
"And to be honest, even setting the doom scenarios aside, I find the utopia scenarios of some industry leaders actually dystopian. And I think a large majority of the public would agree if they paid attention to what they’re saying."
My views on AI have radically shifted over the past few weeks, after spending way too much time reading about it and watching interviews of industry leaders and experts. First, I really love the tools. I spend $200 on Claude every month and the product is worth every penny for me. Probably even more. I think the existing tools can deliver significant increase to future GDP and help us solve major global challenges faced by humanity. However, I just cannot comprehend why the industry is not much more regulated, as its leaders and employees seem to request themselves. I meet a lot of people who have heard of the hugging face incident but don’t know the actual chain of events. If you don’t, you definitely should spend a least an hour reading about it. I also do not know why we need to follow the path that frontier labs are following, to develop ever more powerful models. It seems to me that they are enormous gains that are readily available if we deploy the current technology in economic and scientific organizations without pursuing a super intelligence that they say might kill us and more likely wreak havoc in society. I know a lot of people think these warnings are marketing tricks, and that’s a reasonable assumption. But I don’t see the point of taking such risks. My median scenario is that of many economists: AI will diffuse slowly and lift gdp growth by a percentage point or so, will cause some economic disruption on the way, but ultimately people will get new jobs and maybe work less. However, I am deeply uncomfortable with the absolutely wild level of uncertainty around this prediction. And to be honest, even setting the doom scenarios aside, I find the utopia scenarios of some industry leaders actually dystopian. And I think a large majority of the public would agree if they paid attention to what they’re saying. To their credit, they themselves agree publicly that they should not be the ones making these choices. A large majority of the public is now very skeptical of AI as a source of human progress, in particular among younger adults who use it the most. They should have a say. I don’t want to convince you of anything but to get informed about this topic, express yourself and vote accordingly.
71
For most devout religious people, the person they wished they were is someone who is more holy, sins less, comes closer to attaining true theosis - not someone who abandons their deep faith and all the attendant values because a reddit atheist bashed them over the head with rationalist cognitotherapy for umpteen hours
1
14
272
This is a perfect example of my contention that in the eyes of many CEV proponents, CEV is a tool to coercively misrepresent people's actual preferences as a completely different set of preferences - and this specific case about "God doesn't exist therefore religion-based values are false/irrelevant" is one I've seen previously
3
3
1
31
1,371