Martin Marek retweeted
score centering + masked importance sampling is now the default in TRL's async GRPO Trainer!
Across various staleness and sampler quantization settings, the stability is pretty wild 👇
Tinker's new StableReinforce objective is Reinforce + Score Centering. Here's how I reverse-engineered it 🧵
Off-policy RL is difficult; runs can suddenly collapse because of async RL staleness or numerical mismatch.
StableReinforce is a new method on Tinker that keeps your RL runs stable, even when learning drifts from sampling.
tinker-docs.thinkingmachines…
On a side note, I'm thrilled Thinky has found Score Centering to be extremely stable! From their docs: "In our internal post-training, this loss function has led to extremely stable training, even under highly off-policy async RL and low precision numerics."
Score Centering reference: arxiv.org/abs/2609.20807
Martin Marek retweeted
Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.
• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from scratch in under 4 months
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
Martin Marek retweeted
We reproduced this in TRL's AsyncGRPO 👇
Qwen-1.5B, vLLM in FP8 (weights + KV) and trainer in bf16:
• plain GRPO collapses after ~150 steps, reward 0.56 → 0.10
• ratio caps and masked IS alone don't save it
• score centering + MIS trains like a bf16 sampler (0.72)
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
A cool application of score centering: running RL on heterogeneous hardware
It uses the Miles implementation of score centering, which supports top-p sampling.
Applying score centering to top-p sampling is a generalization of the same "mask replay" correction that Cognition and DeepSeek use for top-p training (under zero TIM, these corrections are equivalent). Another benefit is that under top-p sampling, it is feasible to run exact score centering (which is infeasible with full sampling).
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
Martin Marek retweeted
We've just added score centering to slime!
You can enable sc with --use-score-centering
PR: github.com/THUDM/slime/pull/…
RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem.
We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches — while simple to implement!
🧵 [1/6]
Why is RL so sensitive to training-inference mismatch (TIM)?
With each training step, the trainer is pulled toward the sampler, resulting in accumulating drift.
We exploit this intuition to devise a novel correction method to stabilize RL under TIM, called Score Centering.
Thanks to @m_ryabinin for supervising me on this project during my internship at @TogetherAI!
Our paper includes many more experimental results, intuitions, and derivations.
code: github.com/martin-marek/scor…
paper: arxiv.org/abs/2609.20807
See also @m_ryabinin's thread
RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem.
We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches — while simple to implement!
🧵 [1/6]
Martin Marek retweeted
RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem.
We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches — while simple to implement!
🧵 [1/6]
Martin Marek retweeted
New paper: arxiv.org/abs/2605.26097
The main idea is that we can use an LLM to generate its own replay data to prevent forgetting, as long as we have spare capacity. Very overtrained models have to forget to learn new information.
