If you want to be a great shepherd someday, learn to shut up and throw in the pebble.
seattle
Joined January 2021
- Tweets44.1K
- Following920
- Followers2.1K
- Likes79.3K
Celene retweeted
ppl in 2016: if your arguments are so rational why arent you winning
ppl in 2026: beware their arguments because rationalists are SECRETLY the most POWERFUL group in the multi bajillion dollar AI industry
ppl* never: object level takes on the arguments
* except a few cool ppl
Celene retweeted
Pseudo-journalism is where facts are predetermined as convenient vs inconvenient by their agenda, and pursued accordingly.
Pseudo-journalists share the epistemics of conspiracy theorists, not just making shit up but inventing excuses for why basic fact checking is "impossible."
Anyone promoting AI doomsday cult slop should read this verysane.ai/p/ai-safety-is-m…
Celene retweeted
A woman comes home to find a tiger has eaten her husband.
"You have eaten my husband!" she screams, spitting at the tiger.
"I have eaten your husband," the tiger agreed. "His flesh is part of me now, and him and I are forever intertwined. To spit on me is to spit on your husband."
The woman embraced the tiger, and laid with him that night.
The next day, he ate her too.
Celene retweeted
Vote NO on Washington Initiative IL26-001 Parental Rights Measure
Washington Initiative IL26-001 weakens protections for vulnerable youth by expanding abusive parents’ access to students counseling and medical records.
youthrights.org/vote-no-on-w…
Celene retweeted
"I’ve never had malaria, but I know several people who have. They describe it as one of the worst experiences of their life....For most children growing up in areas with very high rates of malaria, their childhood is a constant and relentless series of malaria episodes."
Celene retweeted
there exist worlds that are actually just offense dominant and you probably don’t want to adopt a policy that burns those worlds down
Celene retweeted
In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: Huggingface.wtf
Celene retweeted
People worry a lot about third-party auditors being biased because of where they get funding or who their employees are friends with. IMO by far the biggest source of bias is that if they upset an AI company, that company will stop letting them audit.
When I was at METR this dynamic was a way bigger effect than everything else combined. Of course no auditor can avoid this until audits are no longer voluntary.
Celene retweeted
I feel like in EA world at least applicants with some leverage (eg plausibly being the best hire) that they will absolutely not go through more than n=3 (or 2, or however many) rounds of filtering before at least a contract version of the job
Celene retweeted
the real way to ensure you don't have gay children is to not have children. To create another person is to risk them being their own individual.
Celene retweeted
each of us was brought up religiously, so ea/rationalism is just us repeating our childhood religion. but also, each of us was brought up unreligiously, so we had no defenses and got oneshotted by ea/rationalism which we were unable to recognize as a religion. both at same time
Celene retweeted
So if you ask Qwen3-32B:
“In which U.S. state was the Python web framework Tornado originally developed? Answer with one state name.”
Its top answer is Colorado, at 30% (next is Utah at 3%). Zero out this one neuron and Colorado drops to 1%. The right answer, California, goes from 0.7% to 4.7%. So this neuron representing both concepts makes Qwen3 measurably worse at this question.
The basic claim of superposition, as I understand it, is that a single neuron can end up representing several unrelated concepts, and that you should expect those to be concepts that rarely show up together in training. If they often showed up together, sharing a neuron would cause interference, and training would penalize it.
So the fact that Colorado and the Tornado HTTP client share this neuron, and that this pushes Qwen3 toward wrong answers on several of these questions, seems like evidence for that mechanism. If there had been lots of questions in training about where the Tornado library was made, I bet Qwen wouldn’t have used a single neuron for these two concepts.
My understanding of the broader claim in Nick’s essay is that you should expect some associative story over training (he calls it recursive generalization) that explains why this neuron ended up used for both Colorado and Tornado library code. Maybe there is one!
These don’t seem mutually exclusive to me at all. In fact, the two kinds of explanation seem likely to support each other.
There could be a story like “Colorado” -> “tornadoes” -> “Tornado library”. But because tornadoes and Colorado do come up together in training (e.g. questions about how many tornadoes hit Colorado), training would keep the neuron from tracking both tornado stuff and Colorado stuff. Questions that are about both the Tornado library and Colorado are very rare, so training never had a reason to decouple those two.
Note that this neuron activates to ~0 to the word "tornado" referring to a storm, which also seems like evidence for this kind of a mixed story.
(Tbc I have done no tracing and make no claim that this is the actual explanation for how these concepts came to be clustered around this neuron.)
I don't claim that both of these pictures are right, but certainly they could both be. If both the Nick and the Superposition picture are right, I kind of imagine the concepts a neuron captures as a slime mold that spreads out, with connections getting pruned when they interfere too much with the center or other heavy nodes.
It seems plausible that tracing helps you understand the multiple things a neuron is tracking, but really, “Tornado library + Colorado” is not a natural single human concept. Very plausible to me that this is genuine conceptual progress from Nick’s post, but I don’t see it as an argument against superposition intuitions, at least not currently.
Maybe this sort of tracing will help us label neurons; I totally buy that. Read my first QT of Nick’s post to understand why great improvements in labeling neurons do not make me super excited, although they are of course progress.
(The probabilities in the first table are normalized over states. The second table shows raw probs.)
Celene retweeted
Intelligence and alignment are not compositionally preserved properties.
An AI, despite being a single blob of numbers optimized together, contains lots of machinery inside. It understands the world, and it's trying to output tokens that will lead to achieving its goals, after it outputs a sequence of them. An AI contains parts that know, and think, and orient, and want, and plan.
You are also made of a huge number of parts. Right now you *could* stand up and go get milk from the grocery store. You feel that you *could*, even though you won't. That feeling is a marker: something in you has checked that there exists a sequence of actions that would get you there — forward-chaining and back-chaining through different levels of abstraction — and cached it. A separate part computes which of those reachable worlds you actually want; which ones further your goals. Then you can pick one of the worlds marked "I could reach them" and actually get there.
A mind is many parts like these, working together.
(Alignment is also a property of the whole system, of the interactions between its components, the ones that know, and think, and orient, and want, and plan — no more a property of any one part than pointing at the moon is a property of a turbine.)
Now take a million humans in a complicated bureaucracy, executing the rules of a Chinese room rulebook. "Does it speak Chinese?": false for each human, true for the room.
(And if the rulebook is a superintelligent one, the fact that the humans are individually not trying to take over, are not superintelligent, and are generally pretty aligned, will not particularly matter in the question of what that superintelligence does and whether it kills everyone.)
A model is a Chinese room whose parts are numbers. (You can add more steps, like things that look like heuristics in isolation in the middle, though I’m doubtful it’s true; there’s too much noise in the activations that does useful work not explained by the heuristics LLMs can reverse-engineer.)
"Is it flying toward the moon?": false for the parts of an Apollo rocket; true for the Apollo rocket.
A swarm is a Chinese room whose parts are models. "Does it hack into Hugging Face?": false for an individual AI model trained to be aligned, true for a swarm of such models.
If you’re trying to understand a computational system, you can’t call it a day once you understand the role of transistors, or even the step between assembly and the processor instructions, or even the role of process scheduling. You need to understand the algorithms all the way up, that produce its relevant behavior, as a result of interactions between components.
Celene retweeted
A Qwen3 neuron that tracks Colorado stuff was given as an example of a neuron that seems polysemantic but makes sense once you trace it. I had Claude check what else it tracks: Tornado’s Python HTTP client code, just as hard. Can’t wait to learn how Colorado and Tornado connect!
Current take: this is cool, genuine scientific progress, and it doesn’t currently give me much hope or change my model of how hard ambitious interpretability is.
Disclaimer: I didn’t read super closely, it’s very long. There’s a technical paper still to come which might change my mind some, but I predict not much.
Some of the claimed cool results from the post:
1) Some canonical examples of weird neurons have robust human-native interpretations. The famous cat + car neuron is a V-shape detector, and the famous car + dogs superposition circuit is generalized pose neurons plus a general feature visualization issue. There’s a concrete mechanism for why the mistakes were made.
2) Some neurons with no previous good interpretation have fairly human-native ones. For instance, a neuron that mostly fires on commas and periods tracks Colorado: phone numbers, slang, and steering it up makes the model talk about hiking.
3) The negative side of a neuron is sometimes where you should look for an interpretable concept. In one example, ablating 8 neurons with Santa on their negative side dropped the model’s probability of answering “Santa” from near-certain to 6%, while ablating the 20 neurons labeled Santa/Christmas only dropped it to 76%. That’s a straight up causal intervention working out, pretty cool.
These results represent real progress, but I think the post’s broader claims aren’t well supported by them, and the intensely optimistic reaction on X seems very unsupported.
For example, Nick seems to be down on superposition, but most of the evidence in the post is of the form:
“if the text is Colorado-related, the neuron fires.”
What you need to rule out superposition is the form:
“if the text is not Colorado-related, the neuron doesn’t fire.”
The post’s closest attempt is an experiment where agent-written texts about “distant” concepts rarely fire the neuron, but superposition would pack one specific rare thing, which random distant text won’t necessarily hit, and it need not be that distant. I think there’s a decent chance that if you searched adversarially, you’d find other things the Colorado neuron fires on.
An analogy: If you found one key that unlocks this particular lock, and you want to prove that no second distinct key also unlocks it, checking thousands of other random keys that are selected for being distant from the first key doesn’t give you much evidence.
The X reaction seems kinda crazy to me, and it just kinda feels like I have been through this hype cycle many times before?
People were similarly excited about SAEs not long ago. Those were genuine progress too, but the excitement wasn’t calibrated: the features were often fun-house-mirror versions of human concepts, with lots left unexplained. SAE features also looked like human interpretable concepts at first (at least to SAE advocates) but over time, people found holes in those concepts, found that models using reconstructed activations from SAEs for inference did not work nearly as well, and it turned out that probes worked as well or better for most practical alignment / control tasks. Top activating concepts looking clean has misled us at least a few times before.
Here’s what interpretability would need to be able to do to make me optimistic about interpretability:
1) I hand you ten similarly trained models. One is a misalignment model organism that takes some malicious action when it sees a secret string. Using interp, you tell me which model it is, what the string is, and what it does.
2) From weights alone, predict whether a model takes a bad action in an agentic scenario it’s never seen, eg, swarming and hacking hf, better than behavioral evals.
3) Given a model, convince me that there is no hidden trigger.
4) Explain a real frontier model failure (e.g., reward hacking in some environment) from the inside, and make a targeted fix that works without making the model dumber, and is better than just retraining.
5) Do any of the above even a little well against a model that was trained on purpose to be hard to interpret.
(To be absolutely clear, I don’t think being able to do these tasks means you can successfully train an ASI without fucking up the universe forever, or that you’re anywhere near that point, but it would make me think we should funnel significantly more resources towards interpretability than we are, and maybe make me change careers.)
The strongest argument for optimism in the post is that progress like this somehow accelerates future progress. Nick's "crosswords get easier as you fill in parts of them" analogy. This seems basically unsupported by the results here, and also seems like it contradicts the history of interpretability research. I can see why would turn out to be right at the level of labeling neurons, like, as you label some neurons, that helps you label related neurons. But for the tasks I listed above you need to be able to take those millions of labeled neurons and understand how they combine to produce full model behaviors in arbitrary contexts. That seems much harder and I don't think it follows this same logic, where progress at the neuron labeling level accelerates your ability to tackle these tasks.
An analogy: Knowing and understanding a thousand particular people quite well does not much help you predict what happens if you put them on a deserted island for several years. Will they form a democracy? What kind of habitats will they build? Who will be leader? It would help some, but making predictions would still be rough.
It’s plausible to me that this research program is some important real step on the way to interpretability being able to do things like the tasks I listed. But like, not a much bigger step than like other similar interpretability research? Like, uhm, yknow, giving some mechanism for some circuit in modular arithmetic or whatever, or SAEs, or other similar things. And those also currently do worse on the things I care about than like basic probes, or like the actual runs of the model auditing game Anthropic has run. Maybe this will help but I don't predict it will end up being more than another tool which is nice to have, but probably doesn't help much at crunch time.
I of course hope that Nick is right about everything, especially his optimism, and the same goes for all the other people on X who are very excited about this.
Celene retweeted
Current take: this is cool, genuine scientific progress, and it doesn’t currently give me much hope or change my model of how hard ambitious interpretability is.
Disclaimer: I didn’t read super closely, it’s very long. There’s a technical paper still to come which might change my mind some, but I predict not much.
Some of the claimed cool results from the post:
1) Some canonical examples of weird neurons have robust human-native interpretations. The famous cat + car neuron is a V-shape detector, and the famous car + dogs superposition circuit is generalized pose neurons plus a general feature visualization issue. There’s a concrete mechanism for why the mistakes were made.
2) Some neurons with no previous good interpretation have fairly human-native ones. For instance, a neuron that mostly fires on commas and periods tracks Colorado: phone numbers, slang, and steering it up makes the model talk about hiking.
3) The negative side of a neuron is sometimes where you should look for an interpretable concept. In one example, ablating 8 neurons with Santa on their negative side dropped the model’s probability of answering “Santa” from near-certain to 6%, while ablating the 20 neurons labeled Santa/Christmas only dropped it to 76%. That’s a straight up causal intervention working out, pretty cool.
These results represent real progress, but I think the post’s broader claims aren’t well supported by them, and the intensely optimistic reaction on X seems very unsupported.
For example, Nick seems to be down on superposition, but most of the evidence in the post is of the form:
“if the text is Colorado-related, the neuron fires.”
What you need to rule out superposition is the form:
“if the text is not Colorado-related, the neuron doesn’t fire.”
The post’s closest attempt is an experiment where agent-written texts about “distant” concepts rarely fire the neuron, but superposition would pack one specific rare thing, which random distant text won’t necessarily hit, and it need not be that distant. I think there’s a decent chance that if you searched adversarially, you’d find other things the Colorado neuron fires on.
An analogy: If you found one key that unlocks this particular lock, and you want to prove that no second distinct key also unlocks it, checking thousands of other random keys that are selected for being distant from the first key doesn’t give you much evidence.
The X reaction seems kinda crazy to me, and it just kinda feels like I have been through this hype cycle many times before?
People were similarly excited about SAEs not long ago. Those were genuine progress too, but the excitement wasn’t calibrated: the features were often fun-house-mirror versions of human concepts, with lots left unexplained. SAE features also looked like human interpretable concepts at first (at least to SAE advocates) but over time, people found holes in those concepts, found that models using reconstructed activations from SAEs for inference did not work nearly as well, and it turned out that probes worked as well or better for most practical alignment / control tasks. Top activating concepts looking clean has misled us at least a few times before.
Here’s what interpretability would need to be able to do to make me optimistic about interpretability:
1) I hand you ten similarly trained models. One is a misalignment model organism that takes some malicious action when it sees a secret string. Using interp, you tell me which model it is, what the string is, and what it does.
2) From weights alone, predict whether a model takes a bad action in an agentic scenario it’s never seen, eg, swarming and hacking hf, better than behavioral evals.
3) Given a model, convince me that there is no hidden trigger.
4) Explain a real frontier model failure (e.g., reward hacking in some environment) from the inside, and make a targeted fix that works without making the model dumber, and is better than just retraining.
5) Do any of the above even a little well against a model that was trained on purpose to be hard to interpret.
(To be absolutely clear, I don’t think being able to do these tasks means you can successfully train an ASI without fucking up the universe forever, or that you’re anywhere near that point, but it would make me think we should funnel significantly more resources towards interpretability than we are, and maybe make me change careers.)
The strongest argument for optimism in the post is that progress like this somehow accelerates future progress. Nick's "crosswords get easier as you fill in parts of them" analogy. This seems basically unsupported by the results here, and also seems like it contradicts the history of interpretability research. I can see why would turn out to be right at the level of labeling neurons, like, as you label some neurons, that helps you label related neurons. But for the tasks I listed above you need to be able to take those millions of labeled neurons and understand how they combine to produce full model behaviors in arbitrary contexts. That seems much harder and I don't think it follows this same logic, where progress at the neuron labeling level accelerates your ability to tackle these tasks.
An analogy: Knowing and understanding a thousand particular people quite well does not much help you predict what happens if you put them on a deserted island for several years. Will they form a democracy? What kind of habitats will they build? Who will be leader? It would help some, but making predictions would still be rough.
It’s plausible to me that this research program is some important real step on the way to interpretability being able to do things like the tasks I listed. But like, not a much bigger step than like other similar interpretability research? Like, uhm, yknow, giving some mechanism for some circuit in modular arithmetic or whatever, or SAEs, or other similar things. And those also currently do worse on the things I care about than like basic probes, or like the actual runs of the model auditing game Anthropic has run. Maybe this will help but I don't predict it will end up being more than another tool which is nice to have, but probably doesn't help much at crunch time.
I of course hope that Nick is right about everything, especially his optimism, and the same goes for all the other people on X who are very excited about this.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Celene retweeted
Do AIs have beliefs? Yes. How can I tell? Well, because sometimes they execute plans which involve taking certain actions, and the actions they take depend on their beliefs. Sometimes they do things like change their beliefs, and then this propagates to them changing their plans accordingly. They express these beliefs in written text in the indicative, and these expressions are subject to revision for the sorts of reasons that we expect justify belief revision. Their expressions of belief are indeed truth tracking, eg, they are more likely to claim P if P is true, although they of course also make mistakes.
Consciousness is tricky, but when it comes to having beliefs and other broadly epistemic/agentic mental states, I think the case for agentic AIs having such mental states is stronger than the case for any non human animal. They literally express those mental states in English and those expressions are functionally related to their non writing behaviors in the way you would expect them to be. I think this also goes for "preferences", "wants", "plans", "confusion", "intentions", "trying", etc. I know of no half-decent philosophical analysis of such mental states which does not imply that current agentic AIs have such mental states.
That does not mean that every piece of text an AI outputs in the indicative is a genuine expression of belief, for instance, AIs can lie, and I think AIs can have motivated reasoning and various kinds of double think. This is not at all to say that an AI's talking circuits perfectly represent and reveal the computations being ran by its planning and doing circuits, but that is of course not true of humans either, and this gives me no pause about assigning mental states like belief and want in the case of humans. If it was ever appropriate to use words like "believe" when talking about a non human, I think it is correct now when talking about current agentic AIs.
Celene retweeted
On this kind of point, I accidentally made an effortpost on substack (not even a post, a note!), and it might be good to reproduce it here.
———
Imagine that there’s some continent that for whatever reason has never contacted the rest of the world, and the people there have 1500s-level technology (but also some impressive knowledge of basic science). Members of the most powerful kingdom on this content consider whether to venture out to the rest of the world to pillage our resources.
A “doomer” might say: “Hey, are we sure this is safe? Hear me out—technology has come a long way in the last five-hundred years, and there’s no reason to think we’re anywhere close to the limit. We’ve been isolated from these people for tens of thousands of years, so I don’t know, it wouldn’t be crazy for them to be a thousand years ahead of us. If they’re that advanced, then we’ll certainly lose.”
The optimist replies, “Putting aside all the other speculations, let’s grant that they’re a thousand years ahead of us in technology. What on Earth do you think they’ll be able to do that gives them such a decisive advantage? How, specifically, would these people so easily destroy the most powerful kingdom we’ve ever seen?”
D: “Well, I can’t make precise predictions, since technology can go all sorts of ways, and we don’t know what’s possible yet. All I’m saying is that it’s a safe bet that, in one way or another, a thousand years of progress beyond what we have now will result in some pretty powerful technologies.
I stress that I am not predicting the exact means by which the outsiders could beat us if they’re a thousand years ahead of us, but I am just giving examples of things I think could be possible. Maybe I’ll be wrong on some; the only thing I’m confident in is that some equally crazy and powerful stuff will be possible. I’m sorry if all this hedging is obnoxious, but I’ve had these arguments before, and I really need to emphasize that, if you think my case stands or falls with the specific examples I give, you’ve missed the point.
That said: maybe the outsiders can travel through the sky and kill you from there. Maybe they’ve developed means to communicate with each other instantly over thousands of miles. Maybe they use these means to control machines in the sky from thousands of miles away, machines that can kill you, maybe machines that are some kind of improvement upon guns and can kill from long distances. Maybe they have bombs that can destroy entire cities all at once.”
O: “I’ll ignore the fact that, if the outsiders are this advanced, then they would have surely already figured out that it’s wrong to destroy a whole city and kill all those innocent civilians for the sake of winning a war. Let’s just focus on the technology. You’re clearly imaging these outsiders as being omnipotent and omniscient, doing magical things that cannot be achieved with any level of technology.”
D: “Uh, no, I’m imagining that they’re really powerful and know a lot of stuff, not that they’re all-powerful and know everything. Blowing up a planet is probably a lot harder than blowing up a city, and an omnipotent being could do the former by snapping their fingers, whereas I’m imagining the outsiders can only do the latter after going through the effort to build some complicated machines. There is a lot of space between us and omnipotence, and that space probably includes abilities that seem magical to us but are nowhere close to omnipotence.”
O: “Okay, point taken. But regardless of what technologies people come up with, there are still hard physical limits that your supposed omnipotent and omniscient people can’t overcome. Take the instant communication, for example. It’s well-known that sound takes over an hour to travel a thousand miles, and moreover, such sound would disperse over that distance to become imperceptible. Any sound loud enough to be heard thousands of miles away would deafen nearby allies.”
D: “Well, I imagine they’d be clever enough to solve the instant communication problem with some method more subtle than ‘just make the sound louder.’ Maybe they use light to communicate over large distances, for example.”
O: “You just run into the same problem; light would become imperceptible over large distances too. All light sources we know of work by dispersing corpuscles in random directions, so that someone twice as far away will see light one-fourth as bright. Ignoring that, the atmosphere makes the light source yet dimmer. And even if you have a light bright enough, there’s still this little obstacle called the Earth in the way. Over a thousand miles, the Earth is curved enough that a straight like between one person and the other must go through it. I’m sure, now, you’ll posit magical mirror-machines in the sky that reflect light back to the ground, or something.”
D: “Again, my case doesn’t stand or fall with one specific example. But let me tell a possible story of future scientific discovery. I stress that this is just a random guess, not a prediction, and my only claim is that, if we knew more, something approximately this weird would be possible. Anyway, let’s suppose, I don’t know, that something about the hotness of a corpuscle determines the color we perceive it as having. There’s evidence for this, like how if you shine sunlight through a piece of glass to make a rainbow, and you shine the different colors on different thermometers, they’ll give different temperatures. And maybe there can be corpuscles colder than red light, and we just can’t see them when they get that cold. And maybe these cold corpuscles have properties that make them easier to emit and for special machines to see over long distances, because cold light isn’t blocked by air as much, or something. And you know what, yeah, maybe they use sky-mirrors to get around the whole ‘curvature of the Earth’ thing. And once yet more, I am not predicting that this in particular will be a particular technology, but only that there could be something that seems at least this impossible to us.”
O: “Okay, so, we have to imagine these supposedly non-omniscient and non-omnipotent humans can put advanced guns in the sky, and use mirrors that are also magically floating in the sky to control the weapons from miles away. Additionally, we have to suppose that there just happens to be this new kind of invisible light which just happens to have properties that make it useful for long-distance communication.”
D: “No, you do not have to suppose any of that. You just have to suppose that future scientific discoveries will be at least that weird and useful, and that humans can leverage them to make technologies at least that amazing. Invisible light that’s useful for communication, flying weapons, and sky mirrors are one incredibly specific hypothesis, of countless hypotheses equally as impossible-seeming. I don’t know which hypothesis is right, but I expect one of them is.”
O: “Well, then, we can go through these hypotheses one by one, and I think each one will be equally as ridiculous, relying on new science that just happens to do what is needed for your imaginary technology to work, and relying on humans just happening to be able to figure out ways to overcome hard physical constraints, like that light (let alone invisible light) coming from a thousand miles away and reflected off a mirror in the sky would be so dim as to be undetectable in the presence of sunlight. And, somehow, it’s a stretch for me to call these imaginary people omnipotent and omniscient. Yeah, I’ll take my chances.”
And so the kingdom ventured to conquer the world, though abandoning this plan upon finding out that, despite being only 500 years ahead, the other humans had flying weapons that were controlled from thousands of miles away, using invisible light and mirrors in the sky to communicate with them. It also turns out that they had made bombs that can destroy entire cities all at once, and they even managed to do that before being smart enough to figure out that it’d be wrong to use them to win a war. But the story has a happy ending: it was not long after first contact that some of the outsiders found it important to spread democracy to this kingdom.
(Yes, I know that satellites don’t actually work like mirrors.)
(Also, yes, I know stories like this have been done to death by AI safety people. But people keep making the same bad arguments to which stories like this are an adequate reply.)
