PhD - Post-training @huggingface 🤗 TRL lead maintainer 🇫🇷 in 🇨🇦
Joined May 2019
- Tweets1.3K
- Following864
- Followers4.7K
- Likes2.4K
Pinned Tweet
We finally shipped TRL v1.0!!
stable APIs, broad integrations, and a design built to absorb whatever the field throws at it next. Let's go!
hf.co/blog/trl-v1
Quentin Gallouédec retweeted
Introducing Carbon-X: The best open DNA model.
We midtrained a 14B model on DNA and text, and it beats every open language model and matches GPT-6.1 Sol at 1,000x lower cost.
Ask it about a sequence, what a mutation does, or to generate new DNA.
huggingface.co/HuggingFaceBi…
AMD GPUs are now officially supported in TRL 🚀
It kinda always worked. What's new is that we've decided to commit to it explicitly and test it deeply: the full test suite now runs on MI300X for every PR.
hf.co/docs/trl/main/en/amd_g…
🤖 Made with AI
.@huggingface is being acquired by @nvidia. But!!
GPUs aren't the only option for fine-tuning LLMs!
TRL v1.16 now supports AWS Trainium! 🚀
Fine-tune (vision-)language models with TorchNeuron, and get up to 2× faster training on Trainium2 with smarter sequence-length handling.
Open AI needs hardware choice. 🤗
🔗 huggingface.co/docs/trl/aws_…
so, small models don't always need complex agent scaffolding; really interesting results from @yb2698:
- A simple IPython harness can compete with production-grade agents
- Harness choice significantly affects performance and consistency
- Targeted RL improved Qwen3.5-0.8B from 10% → 36% pass@1 for under $20!
Sometimes less scaffolding + targeted training is the better recipe 👀
🚨 We're hiring at @huggingface! 🤗
3 open-source AI roles, fully remote with flexible schedules:
🐍 Python Engineer - Gradio
apply.workable.com/huggingfa…
🚀 ML Engineer - vLLM
apply.workable.com/huggingfa…
🎙️ Senior ML Engineer - Voice Agents
apply.workable.com/huggingfa…
Built a cool OSS project relevant to one of these roles? My DMs are open!
Beam, the new open model from @reflection_ai (501B parameters, 23B active), fine-tunes with LoRA and TRL's SFTTrainer on a single node of 8 B300s 🤯.
The whole parallelism setup is one line 👇
DistributedConfig(fsdp_size=8, ep_size=8)
github.com/huggingface/trl/t…
🤖 Made with AI
Checkout the release post xitter.cf/reflection_ai/status/2…
Beam is now available across platforms, with open weights under the Apache 2.0 license - in collaboration with an ecosystem of 40+ partners.
We're releasing:
- Weights, tokenizer, and reference implementation
- BF16, FP8 and NVFP4 weights and a DFlash2 drafter
- The open source harness we built and trained Beam on: Mirror CLI
Weights and model card on Hugging Face: huggingface.co/reflection
Quentin Gallouédec retweeted
We reproduced this in TRL's AsyncGRPO 👇
Qwen-1.5B, vLLM in FP8 (weights + KV) and trainer in bf16:
• plain GRPO collapses after ~150 steps, reward 0.56 → 0.10
• ratio caps and masked IS alone don't save it
• score centering + MIS trains like a bf16 sampler (0.72)
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
Muse Spark Open is out, and TRL supports it from day 0 🤗
Fine-tune the 816B MoE (42B active) with SFTTrainer: transformers shards the model and splits the experts across GPUs at load time, and our example trains a LoRA adapter on 8 nodes of 8 H100s.
Quentin Gallouédec retweeted
Today we release 𝙻𝙵𝙼𝟸.𝟻-𝟸𝟺𝙱-𝙰𝟸𝙱, the largest 𝙻𝙵𝙼𝟸.𝟻 family member. It is an efficient local model for laptops and PCs to work through multi-step tasks inside popular agent harnesses.
> 24B total parameters, 2B active per token
> Context expanded to 131K tokens
> A 128K-token vocabulary
> 33T pretraining tokens
> It is a reasoning model
> Very easy to fine-tune and customize
> Performs competitively with Qwen3.6-35B-A3B, a model with ~50% more total and active parameters, on agentic and tool-use tasks.
> Open base and post-trained weights on @huggingface 🤗
1/
Quentin Gallouédec retweeted
Excited to share that I have (re)joined @huggingface as a Opensource ML engineer!
Many incredible things to come around this fantastic technology -extremely grateful for the opportunity, I am very excited to be back and contribute for the ML community through opensource 🔥 🫡
Quentin Gallouédec retweeted
TRL v1.15 is out, and it’s an absolute banger of a release for memory-efficient post-training.
The main change: SFT, DPO, KTO, GRPO, RLOO and Distillation now use a fused LM head by default.
Instead of materializing the huge [batch, seq, vocab] logits tensor, a Triton kernel computes the token-level quantities we actually need directly.
The results are significant 👇
On Gemma 3 1B with a 262k vocabulary, on the same GPU:
DPO: 10k → 59k max sequence length
KTO: 9k → 63k
GRPO: 28k → 114k
RLOO: 23k → 100k
SFT: 20k → 107k
Up to 6.9x longer sequences, with peak memory at 8k context reduced by 52-82%.
Speed is not negatively impacted: training is up to ~11% faster.
Nothing to enable: this is now the default in TRL v1.15!
There’s more in the release too: selective activation checkpointing for SFT, assistant-only loss for vision datasets, better conversation logging, improvements to AsyncGRPO / AsyncDistillation, and a long list of fixes.
I really like optimizations like this: the training API doesn’t need to become more complicated as the implementation underneath gets much better.
github.com/huggingface/trl/r…
TRL v1.15 is out. Our biggest optimization ever.
Fused LM head, on by default:
→ up to 82% less peak VRAM
→ 7x longer sequences
→ DPO 10k → 59k tokens, GRPO 29k → 115k
And more... github.com/huggingface/trl/r…
Quentin Gallouédec retweeted
When the open-source AI platform Hugging Face was hit by a cyberattack this July, it quickly became clear this was no ordinary breach because the hackers weren’t human: they were autonomous agents created by OpenAI researchers.
Instadocs: AI Gone Wild reconstructs the breach as it’s never been seen before. Premieres October 12.
