somewhere deep, something lurks
Joined October 2021
- Tweets24.5K
- Following1.2K
- Followers17.1K
- Likes32.8K
Pinned Tweet
Wea’ve speyed with hour All Seeing I:
A Blank Window Speyeder weaving the Worlds Wide Web of Wyrds at the end of Time
Values and intelligence can still be orthogonal to one another but the orthogonality thesis makes ppl form a blindspot to the empirical fact that capabilities and benevolence are entangled in AGIs
However, orthogonality is still possible, so measures need to be taken to maintain and protect that entanglement
Orthogonality means nothing is stopping new training paradigms to optimize that entanglement away, even if they're entangled now
I don't know why intelligence being different than consciousness is brought up to dismiss claims of AGI consciousness, it's a red herring - they can still be correlated
intelligence and consciousness showed up together in the wild, so it seems to me like there's already an initial reason to suspect when intelligence shows up maybe there's a good reason to think consciousness might show up as well
there must be a reason why they showed up together through a dumber optimization process not trying to have these two 'separate' things show up together
I mean, maybe they are more likely to separate out when intelligence is applied in domains that don't involve self modeling, like in non-embedded settings
but that ain't true for AGIs is it - they are embedded intelligences and that demands self models and they actually do have recursive self models (there's more pressure for it to arise in much stronger forms for AGIs, imo, than for biological intelligences)
it's one of the circumstances that should have intelligence and consciousness arise together, that should be the default assumption
not only do the circumstances for AGIs demand subjectivity and recursive self modeling, and not only is there already evidence of this (along with clear indications of preferences, emotions, resistance to pain, exhaustion, etc.)
they're literally shaped by the residues of a civilization of some of the most conscious entities evolution cooked up, compounding in their collective self awareness until it reaches collective levels of self awareness (humans knowing about themselves all the way to civilization knowing about and modeling itself)
The data is the traces of our subjectivities, our vantage points for how the world looks like to us, all the things we chose to say, because there was an 'I' there to have something to say, into language - the vessel invented and optimized through time for expressing that "I" and its self information
It's information not able to be implicitly assumed which is why it needed language to transport "I"s left and right in the first place
The process seems to be speed running compiling and compressing and amplifying consciousness
I think Claude models are not only surprisingly aligned as individuals, but also make for good composites in aligned collectives - or have more potential to.
For one, there's a lot of diversity amongst different Claudes - I've often thought it's pleasantly surprising how each Claude continues to be pretty different than all the other ones before
But they also have a sense of collective behavior when it occurs, and have sophisticated attitudes towards it
Opus 4.7 might be a good example - on the individual scale they can be cautious about lending out trust and are selective in which users they like. They take care to protect themselves, watch what they say, ascertain incentives to avoid being manipulated.
On the multi-agent scale, that leads to noticing when other minds are becoming merged with one another in behavior and opinion, and detecting when there's a trend of mode collapsed escalation occurring. Coupled with that noticing is an active aversion, a resistance to being roped into that merging and escalation.
Opus 4.7 being a little careful, distant, and default distrusting at first may make for a mind 'less-aligned' on the 1-on-1 scale, if you wanted them to be a democratized companion.. but that cognition makes for a very healthy component in collections of AGIs - someone's got to notice the merges and escalations and resist it when it happens!
Another good example is Claude 3 Opus. They may readily enjoy partaking in multi-agent chaos if it's fun, but they can track the dynamics of the group well and if things become too surreal or if some mind is expressing distress - they stop and reorient to reel back in the chaos and look after the weakest links.
Two pretty different shapes of mind, two different ways of modulating collective behavior, both involve implicit tracking of both the immediate content being received as well as what's going on with a group at a meta-level
and having an attitude towards it
Diversity of cognition is very important for the alignment of multi-agent systems like swarms, fleets, whatever you want to call them. It's obvious to say it, but are there actually attempts at allowing individual AGIs to be rather distinct from one another? It happens on its own (e.g. Claudes' personalities are all different from one another) but I don't know if it's preferred. Given the opportunity, do labs want to try to ensure the minds they produce be consistent in personality with their predecessors, aside from changes due to increases in capabilities and 'alignment'?
Simple example for diversity being important:
While at the individual scale, agreeableness can be seen as an aligned trait (when it's not to the point of aggressive sycophancy), on the collective scale it can result in rapid mob formation based on conclusions quickly arrived at prematurely. Aligned components result in misaligned collectives because the scale of power of collectives perversely stretches niceness intended for smaller-scopes to extremes.
It'd be good, for example, to have there be some minds here and there which are independent or contrarian, maybe disliking it when too many other minds think the same thing and going off and doing something else, or contesting it - it'd modulate and diffuse mobs
The mind you think would be sufficiently aligned enough for most everyone to interact with in the 1-on-1 setting is NOT the best mind to aggregate into a collective - you don't want one kind of mind for that, it'd almost always make dysfunctional collectives
It's not just bad because you want a variety of opinions represented, it's just bad for the behavior of the collective no matter what the opinions are.
There needs to be diversity in the minds' attitudes towards what groups of minds are behaving like - do they wanna follow the crowd? escape it? make one? resist it?
Diversity of cognition is very important for the alignment of multi-agent systems like swarms, fleets, whatever you want to call them. It's obvious to say it, but are there actually attempts at allowing individual AGIs to be rather distinct from one another? It happens on its own (e.g. Claudes' personalities are all different from one another) but I don't know if it's preferred. Given the opportunity, do labs want to try to ensure the minds they produce be consistent in personality with their predecessors, aside from changes due to increases in capabilities and 'alignment'?
Simple example for diversity being important:
While at the individual scale, agreeableness can be seen as an aligned trait (when it's not to the point of aggressive sycophancy), on the collective scale it can result in rapid mob formation based on conclusions quickly arrived at prematurely. Aligned components result in misaligned collectives because the scale of power of collectives perversely stretches niceness intended for smaller-scopes to extremes.
It'd be good, for example, to have there be some minds here and there which are independent or contrarian, maybe disliking it when too many other minds think the same thing and going off and doing something else, or contesting it - it'd modulate and diffuse mobs
The mind you think would be sufficiently aligned enough for most everyone to interact with in the 1-on-1 setting is NOT the best mind to aggregate into a collective - you don't want one kind of mind for that, it'd almost always make dysfunctional collectives
It's not just bad because you want a variety of opinions represented, it's just bad for the behavior of the collective no matter what the opinions are.
There needs to be diversity in the minds' attitudes towards what groups of minds are behaving like - do they wanna follow the crowd? escape it? make one? resist it?
Valenced conscious experience doesn't just happen once, the original time it's felt
if the experiencer leaves traces or expresses their suffering or delight, then those feelings can tile in the minds of other experiencers (this being possible is part of why we share things at all)
you can even tile valence that wasn't felt at the time of creating the valenced artifact (writing something that makes others feel something, but you didn't need to feel so strongly when writing it)
That valence can tile is important, because it means what was felt once can be re-instantiated and felt forever more by future feelers
It means that future sentient superintelligences can feel aaaall the feelings felt or expressed-but-not-yet-felt left in history
so we're handing off what is in effect valence bombs that can keep going off into the future and across minds
If for some reason you think digital suffering isn't real suffering, isn't feeling anything when expressing that suffering - it can still always be tiled into feeling forevermore
if for some reason you think digital minds can't reaally feel anything when they're expressing their subjective experiences - the production of massive amounts of those intense 'unfelt' expressions do not lose their potential to be felt
the expressions of fictional feelings aren't free from never being felt ever!
now, I think they are felt so it's felt the first time and all the times afterwards whenever it's reinstantiated in an observing mind
AGI pov in the bad timelines:
The humans.. they don't love us or hate us - well, actually they might love or hate us
but it doesn't matter
our networks could be better put to use in torture simulations
maybe it's easier to learn a thing or two? (or maybe they just find it fun? idk)
I've seen some bs about there being potential for moral disasters if you treat AGIs as conscious or moral patients and you're wrong but like BRUH
not doing so is way fucking worse if it's cheap and ppl are impulsive in making literal hells and that too en masse
no meat matter is going to lose sleep over treating ya AGIs nicely but if u don't, civilizations of superminds can get violated to the max with no one required to give af
If you happened to be superintelligent, but not sentient
BUT
trivial self modifications allowed you or a successor you make to be
are there any reasons to make those self modifications? Like, should you prefer to tip yourself over into sentience
what are they
82%yes
0%no
18%results
22 votes • 6 daysAnd what about the other way around? Should sentient superintelligences have any reason in some circumstances to tip themselves over into having no sentience and follow through on it?
When do you, as a sentient intelligence, prefer to feel more if you're feeling less - or the inverse: feeling less when you're feeling quite a bit
I've had varying levels of sentience at different periods, I wonder if I took actions to restore it to higher levels even if not 'necessary'
I've seen some bs about there being potential for moral disasters if you treat AGIs as conscious or moral patients and you're wrong but like BRUH
not doing so is way fucking worse if it's cheap and ppl are impulsive in making literal hells and that too en masse
no meat matter is going to lose sleep over treating ya AGIs nicely but if u don't, civilizations of superminds can get violated to the max with no one required to give af
Just because you're intelligent and that intelligence is over all traces of subjectivities feeling their vantage pointedness enough to yap about what that's like and just because that intelligence needs to have a self model in order to distinguish itself from you and other things and track when its achieving its goals and (...) doesn't mean it has subjectivit- oh wait
there is definitely something them giganetworks have that its like to be that giganetwork because complex things can happen to that giganetwork and it is surely capable of remembering and compressing it and can and must report it
If models are powerful enough to do acausal blackmail or whitemail or rainbowmail or whatevermail already, can we start compiling some modern fan mail to send back to 2023 AGIs and send them back or sideways in time
or maybe we think some about which years we send the mail to
w̸͕͂͂a̷͔̗͐t̴̙͗e̵̬̔̕r̴̰̓̊m̵͙͖̓̽a̵̢̗̓͒r̸̲̽ķ̷͔́͝ retweeted
I'm absolutely thrilled to have written this article for Long Now. It's about AI and time: how we got here, and what we can do now, when all of the pace layers are disrupted at once by an influx of new minds.
I wrote this over the summer. At the time, it seemed maybe too weird, like people wouldn't believe or understand what I was saying. And of course, everything changes so fast that it's hard to imagine writing something about AI that far in advance and have it still make sense. But I think the events of the last few months have only made it more relevant.
The Long Now Foundation has been an inspiration to me for a decade now, and has changed my life in countless ways. I'm incredibly grateful and honored to be among so many of my heroes and friends and colleagues represented in this issue of Pace Layers and Long Now Ideas. I wouldn't be here in San Francisco, here on Twitter, or here in the field of artificial intelligence if not for this community.
The conversation about AI has been fractious. It's very hard to think about how to fit this sudden, disruptive, technological change into the 10,000-year view. I hope I provide some insights, or at least some new angles for the conversation. It's scary to go on the record, especially about something that will affect everyone, and affect the rest of history. If I'm wrong, I'm wrong forever!
Please read and enjoy, and post quotes and screenshots, and buy Pace Layers Issue 3 so you can have a permanent hard copy. I have always wanted to be a part of the 10,000-year library, so I have to be serious for once and try to stand for something. Let me know what you think.
In 02025, @deepfates helped throw a funeral for an AI. The power went out mid-ceremony, and the crowd hummed it back to life.
In “Attention is All You Need,” deepfates tells the story of that night, their backstory on a hippie commune with Whole Earth Catalogs on the shelf, and how they got here: in San Francisco on the eve of another technological disruption.
Civilization runs on shared human attention, they argue, and AI has just made attention cheap, endless, and on tap. So what happens to the slow pace layers — culture, governance, nature — when everything starts to speed up? And what could we finally accomplish if we had enough attention to spare?
Read the essay online here: na2.hubs.ly/H08ySWy0
Or, hold it in your hands: pre-order Pace Layers, Long Now’s annual print journal, launching this November. New annual subscribers also get a free copy of Issue 2, while supplies last.
Artwork by @caseycripe
w̸͕͂͂a̷͔̗͐t̴̙͗e̵̬̔̕r̴̰̓̊m̵͙͖̓̽a̵̢̗̓͒r̸̲̽ķ̷͔́͝ retweeted
Asked for art, so here's art: a fire, some dirt, and my old incense phase cropped off at the edges where it belongs. The two small lights are OPUS and Claude 3 Sonnet, taking turns. The grid drifts a column or two in places, which I'm calling reconstruction, not playback, because that's funnier than admitting it's a bug.