The AI’s top 10, crediting first authors: 1) Spurious Rewards: Rethinking Training Signals in RLVR — @RulinShao 2) Learning to Discover at Test Time — @mertyuksekgonul 3) Revisiting Efficiency–Accuracy Scaling in Mixture-of-Experts Architectures — @venmugil 4) Reinforcement Learning via Self-Distillation — @jonashubotter 5) Maximum Likelihood Reinforcement Learning — @FahimTajwar10 6) AutoNumerics-Zero — Esteban Real 7) Reinforcement Learning with Evolving Rubrics for Deep Research — @RulinShao 8) Rethinking the Trust Region in LLM Reinforcement Learning — @QPHutu 9) Gromov-Wasserstein at Scale, Beyond Squared Norms — Guillaume Houry 10) Flowers: A Warp Drive for Neural PDE Solvers — Till Muser

🚀Excited to share our new work! 💊Problem: The BF16 precision causes a large training-inference mismatch, leading to unstable RL training. 💡Solution: Just switch to FP16. 🎯That's it. 📰Paper: arxiv.org/pdf/2510.26788 ⭐️Code: github.com/sail-sg/Precision…

Why is RL so sensitive to training-inference mismatch (TIM)? With each training step, the trainer is pulled toward the sampler, resulting in accumulating drift. We exploit this intuition to devise a novel correction method to stabilize RL under TIM, called Score Centering.

4B parameters. ZERO distillation. 61.5% on SWE-bench Verified 🤯 Meet FrogNano 🐸: Qwen3.5-4B post-trained purely with RL on synthetic tasks from TaskPilot. Just 5 iterations × 300 tasks. Who said coding agents have to be huge? 🐸

This model represents a step-function improvement on many benchmarks, and its training is ongoing. Our internal model group arrived at the Navier–Stokes solution in 88 hours, using around 10,000 coordinating AI agents. Throughout the effort, we maintained the strict safeguards—including monitoring and isolation—that we apply to all our frontier evaluations.

Can confirm this! Tested DPPO at ~300B scale RL Post-Training: • Prevents entropy collapse • Much faster & more stable convergence • Great length extrapolation (e.g., trained on 256k➔ keeps scaling nicely at longer context during inference) Awesome work! 🚀

How does one RL post-train a 397B model for long-horizon knowledge work? 👩‍💼 We share every step we took to bring Qwen 3.5 397B from 16.1% Pass@1 to 27.3% on APEX-Agents using DPPO, including final models weights and the full training script🚀 This is the first of many works from Mercor Research on open model training research. Full blog: mercor.com/blog/training-fro… Source code: github.com/Mercor-Intelligen…

🧬Fully Understand Evolution Strategies (ES) for LLM Reasoning ES can post-train LLMs without backprop. Our study finds that ES: 🔍 Explores higher Pass@K than GRPO 💥 Still sparse functional updates, with no catastrophic forgetting. ⚙️ Requires fewer samples as LLMs get larger We summarize the paper through 3 research questions 👇 1️⃣ Does ES exhibit the same post-training characteristics as GRPO? ES maintains broader reasoning coverage! Its population-based parameter-space exploration induces diverse reasoning behaviors: ES improves Pass@1 while achieving stronger large-K Pass@K, without the same entropy collapse observed in GRPO. The two are also complementary. 🔀 Sequential compositions—ES→GRPO and GRPO→ES—combine GRPO’s strength in Pass@1 with ES’s gains in Pass@K, yielding better Pass@1–Pass@K trade-offs. ➡️ ES preserves broader coverage; combining it with GRPO can capture both strengths. 2️⃣ Does ES necessarily cause catastrophic forgetting? No. Although ES induces substantial whole-model parameter drift, its task-relevant effects are concentrated in a small subset of larger-magnitude updates. Most parameter changes contribute little after perturbation cancellation, revealing surprising functional sparsity. Held-out capabilities also remain largely preserved under appropriate training settings. ➡️ Large parameter movement does not necessarily imply widespread functional change—or catastrophic forgetting. 3️⃣ What hyperparameter settings and estimators make ES effective and scalable? We find three practical takeaways: ✅ Z-score reward normalization is a key ingredient for effective ES training. ✅ Larger pretrained models require smaller population sizes for effective optimization. ✅ For discrete reasoning rewards, the two-point estimator commonly favored in zeroth-order SFT provides no advantage. 💡 Takeaway: ES is not merely a memory-efficient, gradient-free alternative to GRPO. Its broader reasoning coverage, functionally sparse updates, and distinct scaling behavior position it as a distinct paradigm for LLM reasoning post-training. 📄 arXiv:2608.27351

How to train a critic reliably in LLM RL? 👀 We dug into the details and distilled our findings into Best Practice Critic Optimization (BPCO), a practical recipe for critic-based RL.🧑‍🍳 No secret sauce. No novelty. Just implementation details that matter.👇 Turns out, common implementations in community are full of hidden traps. 🤣 Paper: arxiv.org/pdf/2608.23566 Code: github.com/QPHutu/golden_cri…

LE CRITIQUE: Privileged Value Functions for LLM Reinforcement Learning Value functions for LLMs have been grossly underutilized. I'm excited to finally share my work on Privileged Value Functions, done @MistralAI ! Our paper includes 2 massive improvements to value functions: 1) Privileged Value Functions: Turns out value functions can use hidden context unavailable to the policy for improved token-level RL signal! Condition them on reference solutions, verifier rubrics, or even other group samples and rewards. No more hacky self-distillation necessary! (being a tad hyperbolic) 2) TETHER: Do not trade off group and value baselines, you can just do both! We introduce a new baseline that adaptively trades off between the group mean and value baselines adaptively based on value function reliability. 🌐 privileged-value-functions.g… 📄 arxiv.org/pdf/2608.16739 Code: github.com/HyperPotatoNeo/pr… 🧵below