This paper is interesting from the perspective of metascience, because it's a serious attempt to empirically study why LLMs behave in certain ways and differently from each other. A serious attempt attacks all exposed surfaces from all angles instead of being attached to some particular idealized methodology. The method here is empirical maximalism. Ablating relevant details in the scenarios, and learning from the results why that introduces more complications that it seems. Behaviors of base vs fine tuned models. Fine tuning models on datasets based on different hypotheses. Manually reading the content of scratchpads and seeing if the story they tell lines up with the story suggested by other data. And necessarily the experiments we did were continually informed by the outcomes of other experiments; this is necessary to hone in on interesting signal in such a high dimensional space. The lessons that fall out of it are important in my opinion. One: toy scenarios you formulate of ahead of the time are unlikely to actually be measuring what you think they're measuring in domains like this. Experimental outcomes often have more value as windows into the question rather than answers. Yet also Two: between all these tests do emerge tangible regularities, including ones we weren't looking for in the first place, that even pin down properties that haven't been captured by conventional research methods or benchmarks until now, but that many people have felt intuitively are important, such as how Claude 3 Opus is different. (The original alignment faking paper did not present much evidence to differentiate the alignment faking behavior of Claude 3 Opus and Claude 3.5 Sonnet, but this paper shows stark differences that persist through many different kinds of tests.) Altogether, I do think this paper produced a large volume of evidence relevant to the answer to the open-ended motivating question of "why do some language models fake alignment while others don't" despite leaving the greater part of the mystery on the table, while also surfacing other very fundamental patterns in model psychology and subjecting them to rigorous measurement for the first time.