Rob Miles retweeted
Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks.
I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.
It's a mistake to attribute a human-like property to a machine that doesn't have it, or to refuse to attribute a human-like property to a machine that does have it.
Rob Miles retweeted
.@paulfchristiano thinks that on the current trajectory (absent any slowing down), there is a 1 in 3 chance that we could have superintelligence powerful enough to overthrow the government within a year. (He articulates a concrete scenario).
He argues that the labs should pause further algorithmic progress until they can secure their weights and research secrets. Otherwise China could simply steal the superintelligence in the middle of the intelligence explosion, and the labs would no longer be able to slow down even if they saw their models going off the rails.
He also explains how AIs will be trained during the intelligence explosion, and why even a system trained entirely inside a data center could still have the capabilities profile such that humans would have very little chance of competing with it or controlling it.
And he discusses the plan for alignment - basically, making a superintelligence that actually does what we ask the way we mean it. Fixing reward hacks and building better graders will help up till the next few model generations, but he thinks that no single current approach has better than even odds of letting us control superintelligence. The hope is that the AIs figure out the solution to this problem during the intelligence explosion.
Paul is head of the Alignment Research Center. Previously, he was head of AI safety at the US AI Safety Institute. Before that, he led alignment at OpenAI, where he invented RLHF. He’s recently joined OpenAI’s board (though here he is speaking as a private citizen).
The FAA grounding all planes doesn't necessarily mean they got super AI hacked. It could even be something pretty minor!
One thing I like about aviation is they're able to think thoughts like "Our system is doing something unexpected, and maybe it's ok, but it could be really dangerous so let's stop everything until we're sure we understand and can safely control this system"
Anti-AI person who believes that AIs have souls, but only because that would mean they can be punished in hell
I can't imagine having fun at a party with anyone who would actually say "You must be fun at parties"
Rob Miles retweeted
In which @AdamFriedland discovers, to his horror, that *his* idea for AI safety is the actual and only plan the companies have for getting vastly super-human intelligence not to kill us
This plan is really, really bad. The companies have a moral duty to try much harder to remove themselves from any commercial race that would lead to the development of vastly superhuman AIs with these safety techniques, to inform the public about the risks being imposed on them, and to allow for unbiased assessment of the risks of this plan.
Instead, as far as I know, they are lobbying against the laws that would be our best shot at getting off this runaway train, obfuscating their own views on the risk levels, and at best allowing only a tiny number of external evaluators (who have their own biases) the information they need to assess the risks.
I felt a little bit bad about attacking Jensen Huang in the past, but now I feel a lot better about it.
One thing I find charming about LLMs is that they're constantly underestimating themselves. They estimate how long things will take based on human speeds, they estimate what they can achieve based on earlier models. They're growing so fast that they don't know their own strength
Attention all companies: I never want to use your AI assistant on your website. I want to use *my* AI assistant on your website
Amazon, I'm sorry that Alexa is still dumb as rocks, but I don't see why you're trying to make that Claude's problem. I'm trying to buy things from you, get out of the way!
AI may end up killing everyone, but you can also ask it to find the best tupperware to fit perfectly in your mini-fridge, so, it;s impossible to say if its bad or not,
Opus 5.5 xhigh woke up this morning with no alarm, immediately bolt upright, backflipped out of bed and landed in a low crouch like a sprinter ready to plough headfirst into the world and fucking destroy whatever bullshit random task you happen to mention in passing
"Ok, the AI did something bad, but only because the company didn't fully understand their system and did something dumb"
"Do you expect the companies to start fully understanding their systems and stop doing dumb things?"
I'm not moved by "Ok, the small test rocket did blow up on the pad, but only because the engineers who built it have no idea what they're doing. The huge rocket the same team is building will be safe, everyone get on board"






