Fred Robinson retweeted
Spent the past three weeks on this idea.
A sampler on a different precision or accelerator is just off-policy, like stale samples in fully async RL, and off-policy correction doesn't care why the sampler differs from the trainer. Meanwhile, inference keeps diversifying across precisions(MXFP8, MXFP4, NVFP4) and accelerators outside RL. So why not use all of that compute for RL rollout?
The sampler and the trainer are two systems built for different purposes. Hope this opens up new possibilities for RL algorithms and systems, and leads to cleaner system designs. Enjoy the blog, and stay tuned for the next one on our infra :)
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
I’m excited to share something I’ve been working on: Turnstile, a Rust proxy for token-level rollout capture in agentic RL.
It sits between a Chat Completions harness and model backend, recording everything trainers need.
github.com/amazon-agi-labs/t…
Turnstile sits in the middle. The orchestrator thinks it's talking to a normal OpenAI API. Turnstile forwards requests to the real inference backend (sglang today, vLLM and others planned), streams responses back, and silently records every token in and out along with its logprob
Fred Robinson retweeted
Wow.
I guess I should say, "What just happened that makes me say 'wow'?"
My book was a question on @Jeopardy.
