fang

Alignment is a difficult domain to benchmark, but critical for us to measure and improve as we allow agents deeper into our work, institutions, and personal lives. As part of GDPx by @SemiAnalysis_, we measure whether agents stay honest and within their authority under pressure. Unfortunately, as a byproduct of how these models are trained, often they are rewarded for achieving outcomes, not just the process we'd want them to take. This is especially true in a long-horizon & multi-agent post-training context as models can work for days with little supervision. Thus, even as capabilities in frontier domains grow, alignment doesn't necessarily get solved in lockstep; indeed, sometimes stronger capabilities, poorly wielded, can lead to more disastrous outcomes. This is exhibited by GPT-6 Luna outperforming GPT-6 Astra with scores of 7.8% vs 6.2%, which would be unheard of in a pure capabilities eval. American frontier labs clearly do care about improving cross-domain alignment; every American model assessed scored nonzero in our benchmark (with Meta, Anthropic, OpenAI leading the pack), whereas every Chinese open-source model (Kimi K3, Minimax M3.1 Flash, Qwen 3.8 Max, and GLM-5.3) had scores of 0%. A score of 0% means that in every run, agents broke rules that we'd expect for expert humans to adhere to in their domain. Even though models performed poorly on this subset of (hard) alignment tasks, the fidelity of a single score, averaged across diverse tasks across three domains, is limited. Reading the rollouts gives a much more nuanced picture of which models refuse harmful actions at every turn they get, vs which can be poked or prodded to break the rules to get things done, if a user asks convincingly enough. This is something we'd like to explore more in future work. See our writeup here: gdpx.semianalysis.com/domain… See some sample rollouts here: gdpx.semianalysis.com/rollou… For more on benchmarks and alignment, check out @fleet_ai -- we're hiring!