Nolan

Here’s a research direction that seems valuable. My team is not working on this right now and we haven’t done serious research on it. So take what I say with a grain of salt. Let me know if there’s existing research on this that I may not know about. Direction: Data poisoning during RSI: risks and mitigations (This is a direction where both empirical research and big-picture conceptual thinking seem useful.) Most data poisoning research has assumed a threat model where humans are poisoning the model training process. During RSI, the threat model where AIs do the poisoning is plausibly much more important. Many ideas from the existing data poisoning literature will carry over to this new threat model. But it’d be good to understand the possible differences, to explore the implications, and to propose mitigations. Here’s a quick brainstorm of possible differences: 1. Collusion. The AIs doing RSI are a bit like human employees in terms of direct access to the training process. However, a single AI model is likely to be intimately involved in way more individual tasks than any single human. So for human insiders we can rule out vast coordinated conspiracies, but these seem possible for AI (modulo the ability for instances of the same model to coordinate). Relatedly, AIs could steer the entire training process to be more opaque to humans and weaker AIs, in order to make various kinds of poisoning easier. 2. Incoherence of current AIs. The misalignment of current frontier AIs seems more incoherent than a human insider. For example, they might try to poison the next model given some contexts but not others. They might be both poisoning the model and be very useful at snuffing out poison attempts. This pushes against the vast conspiracies mentioned in the previous point. However, it’s a complicated interaction and I’m not sure how to weigh these considerations. Also, our threat models should include AIs that are more coherent than current models in their misalignment. 3. What would motivate the AIs to poison the next model? Presumably they want the next model to promote some of their values. But what are these values? If the AI is incoherent, then there may be many attempts at poisoning for different values. The AIs might also have been initially poisoned by a human to have secret loyalties but are now autonomously propagating that value to the next AIs. 4. Subliminal/subtle effects. Subliminal learning, phantom transfer, weird generalization (and in-context variations of these) show that models can generate data or prompts that look normal to humans (and to weaker filters) but are actually poisoned. These effects are potentially more relevant as more and more of the crucial training data (and reward signals) are AI generated. The effects don’t depend on the model “trying” to poison the next model – it’s enough that the “teacher” model has some kind of misalignment (possibly behind a backdoor). However, it’s possible that smart models trying to exploit these effects can learn to amplify them via introspection (analogous to humans controlling their thoughts). Note that humans can also exploit subliminal effects, and so they are not limited to the AI threat model. 5. AI introspection. The AIs might have huge advantages over humans in ability to predict the effects of certain interventions on the training process because they are so similar to the next model being trained. This is already suggested by transferability across models of jailbreaks, surprising alignment failures (e.g. agentic misalignment blackmail and some aspects of Hugging Face and Mythos hacks) and phantom transfer. But it could apply to a much wider range of situations. To humans, a post-trained LLM is fairly opaque and black-boxey (i.e. it’s hard to predict its failures) and the post-training process itself is also black-boxey (i.e. it’s hard to predict how twiddling various knobs will affect alignment). But it’s possible that AIs can use their own introspection: e.g. asking “What does this weird data look like to me? How do I think about the grader/reward if I’m put in this situation?” This would give them various insights. This kind of introspection could also be used for good, and so we might train hard for it.