WaterBucket

My biggest takeaways from @AnthropicAI interpretability researcher @mlpowered: 1. Predicting the next token requires understanding the universe. Language models got better than humans at predicting the next token around 2019. Once a model masters the easy tokens (sentence structure, then basic facts), the only ones left are hard, like the final paragraph of a PhD thesis. To finish “The capital of the state that contains Dallas is . . .,” a model has to know that Dallas is in Texas and that Texas’s capital is Austin. 2. LLMs plan ahead, even though they write one token at a time. This was one of Emmanuel’s team’s most surprising findings. When a model writes poetry, it doesn’t stumble into a rhyme. Looking at intermediate thinking states during poetry generation, researchers found that as soon as the model finishes the first line, it has already picked the word that will end the second (“rabbit”), then refers back to that plan for every word after. 3. Anthropomorphizing AI can often be a useful approach when trying to predict how they will behave. Models learn from human text and develop representations of the world from it. Training on millions of dialogues is like training an actor to play every part, and your chats with Claude are literally formatted as a script (“Human: . . . Assistant: . . . ”). In Emmanuel’s words, “Something that’s mimicking a human is really close and is much closer than a program.” 4. Interpretability research looks more like neuroscience than software engineering. The job involves taking a system you did not design and running experiments to infer why it behaves as it does. You examine seemingly meaningless lists of numbers, pass the model huge volumes of text, and watch for patterns that activate from different inputs—not unlike the methods used in brain imaging studies. “A lot of what we do feels so much closer to biology than to engineering." 5. Models can tell you about only a small slice of what they’re thinking. Emmanuel’s team found that just a small subset of a model’s internal representations, which they call the “J space,” can be reported back if you ask the model what it was thinking about. They compare it to a theory of the human mind where most processing happens below the surface. In one experiment, they swapped “Spanish” for “French” inside the J space. Asked what language a passage was in, the model said French. Asked to simply continue the passage, it kept writing Spanish. 6. Studying models closely often raises more questions than it answers. Emmanuel notes that the confident claims that LLMs are “just X” or “nothing more than Y” almost always come from people who haven’t studied their internal workings. His own experience is the inverse: studying the models closely has increased his uncertainty, not reduced it. The complexity that emerges from training is consistently underestimated from the outside. “I find that the more you look inside, the less certain you are about how they work.” 7. Even the people who build AI models don’t fully understand how they work. As Emmanuel describes it, models are grown, not built. “To make them is not to understand them.” Engineers write the code, pour enormous amounts of data through it, and a model’s capabilities emerge. That’s why his team exists. Their job is to open the model up and figure out what it’s actually doing between your prompt and its answer. 8. Interpretability as a field is small, young, and open to newcomers. Emmanuel estimates the field at hundreds of researchers, split between major labs and academic groups. Because it is so new, getting started doesn’t necessarily require deep prior expertise. He points to platforms like Neuronpedia, which hosts models and lets anyone run interpretability experiments directly. “It’s such a new field that the on-ramp is mostly self-studying and reading some of the key papers. There’s so much to explore that if you’re at all curious, you should just give it a try.”