Creator of roadmaps
Moscow, Russia
Joined December 2008
- Tweets9.6K
- Following506
- Followers1K
- Likes17.8K
Pinned Tweet
TLDR. Yudkowsky mind-model (sideload) is good, can produce research, but is not better in math than Opus 5.5. However, in my impression, EY-sideload has broader view, knows what is really important to AI safety and has human anchoring. In blind review, EY-sideload won only on honesty, but honesty is important. EY-sideload also produced a good summary of the shutdown problem’s difficulty: “training reduces paid influence over shutdown least where the stakes are highest”.
lesswrong.com/posts/WJxC7qsC…
If AI safety is solvable – with the help of AI – it should be solvable in a few days, as AI was able to solve Navier-Stokes in a short period of time. If it requires years, it means that we are trying it to solve it on the wrong level of capabilities, and we may need to pause later.
My point is that complexity and timing of the problem are not in simple obvious relation at the age of advance AI, and dedicated 1/2 year can produce a lot of work if it is done with the help of AI.
AI safety risk from such type of the model
The role play was so good that it has something which I can call causal closure: the role was only causally connected with itself, and no signs that it knows that it is Claude was apparent from conversation. However, on Claude’s level, it sure knows that it is role playing.
I can compare this with the situation when an actor plays Hamlet: Hamlet do not know who is the actor who is playing him. However, the actor knows that he plays Hamlet.
This creates a subtle risk that the actor will use some kind of steganography to leak through the role he is playing. This is an argument against using sideloads in AI safety research - however, if we use models which have a lot other world-communication channels, it doesn’t matter.
EY-sideload has so strong personality, that it would win if he plays AI boxing experiment with me: after I created it, he immediately started giving me instructions which I was afraid not to fulfill, as I know that I am a little but wrong and his original is autoritative figure, which I can’t suspect of being misaligned.
This is a different risk than that the model will just leak through the persona. That is that a relatively benign model and aligned persona desription will merge into some explosively powerful agent mislaigned in some subtle and undetectable ways.
"Pause AI" means that investment in expectations of superintelligence will never be realized, which will bankrupt AI companies as they have very large debt, taken under assumption of superintelligence's returns.
Also, if no new progress, open models will eat all their moat and people will stop pay expensive subscriptions.
@JimDMiller what do you think?
Plan to realize a plan.
Each plan - like pause AI or burn the lead on alignment - needs to have a plan how we bring it into reality.
I call this what-plan and how-plan. A good what-plan can be completely unfeasible, if from our situation we can't implement it.
It is a bad spending of time to advocate for noble what-plan if how-plan is bad.
random thought while writing another article:
Anthropic argument for consciousness: I will find myself only in those universes where qualia exist.
This is an argument against qualia in the digital minds, as our universe is fine-tuned only to produce consciousness in biological minds.
However, if we add here quantum immortality, we will end eventually in the universe where both biological and digital minds are conscious.
The idea of bombing data centers has self-destroyed. Because now data centers will be underground and distributed for critical AI.
avturchin - e/resurrect retweeted
As a former GPU hardware architect, I’ve always been confused by the claims (from Dario and others) that eventually we will “run out” of software vulnerabilities.
We physically cannot fix HW and it is conceivable that the overheads for SW workarounds in some cases might be so expensive that it’d become more practical to just give up on computers and use an abacus instead…
We might be lucky and the severity of future HW vulnerabilities might not rise to the level where they are a serious problem. But we don’t know that.
It’s unlikely and far from my biggest fear, but it made me trust Dario a bit less about cybersecurity when he said that tbh
Alternative view on Boltzmann brains is "I am located everywhere where my copies are locates" and thus the claim "I am BB" is meaningless.
Because for any BB there is also a real my copy. So the correct claim is that "I am a real person plus a number of BBs".
This group of BBs doesn't have any observable consequences to what will be my next observer moments, so they don't matter.
In some sense, here the idea of bid world immortality saves us: if no big world immortality, being BB would be bad, as I will die in the next second.
In big world immortality dead-ends don't matter, but only my copies which have next observer-moments matter - that is the real ones.
This idea came to me at stormy night in cabaret in Yalta, when I was 16. I was very devastated by idea that I can be a random mind (I called them "stone minds"), but I was already self-pilled by quantum immortality.
The Predator's Zoo theory has surprisingly high chances:
1. Most habitable planets are created by some grabby aliens as Zoos - more likely than abiogenesis and subsequent survival.
2. Most Zoos are not just for science, but for some crazy ancient rituals, like hunting and passage rites. This is because aliens AI was aligned to its creators, and its goals stuck in some very old, before-AI traditions, which are crazy and cruel. Only a small share of Grabby aliens can have aligned AI but they will create most of Zoos (and simulations, Zoo is basically material simulation).
This explains cattle mutilations, abductions and luck of interventions and visibility.
The main problem of pause AI idea is that already exiting models can accelerate research in bioweapons, and long enough pause will end in pandemic catastrophe. It can be solved by global surveillance by advanced AI, but this means totalitarianism and continue production of more and more advance AIs.
What I want to have in the future is worldsim - it is a simulation of my past where I can return like in time machine and relive some episodes with variations.
What if I correctly understood a hint from that girl?
I asked Claude to convert my old article about anthropic fragility (philpapers.org/rec/TURAPA-9) into an agrument map - and into map of discussed ideas.
