First Blogpost! Late stage patching full-attention transformers to Gated DeltaNets (GDN) Can we convert off-the-shelf pretrained full-attention transformers into hybrid Gated DeltaNets (GDN) without sacrificing long-context performance? Existing distillation baselines restore Common Sense Reasoning (CSR). Some of them also recover a non-zero long-context retrieval but score 0 on long-context reasoning benchmarks like AIME without expensive continual pretraining. Introducing OPAL — On-Policy Attention Linearization 🚀 Our initial results show that, using just 3B training tokens, we can convert MiMo-7B-RL-0530 into a 3:1 GDN-to-full-attention hybrid while recovering up to 93% of its long-context reasoning performance and 100% of its long-context retrieval performance, evaluated at context lengths of up to 32K tokens. hunchings.github.io/late-sta…