Congliang Chen

🚀 Excited to share our work at Bytedance Seed! Knapsack RL: Unlocking Exploration of LLMs via Budget Allocation 🎒 Exploration in LLM training is crucial but expensive. Uniform rollout allocation is wasteful: ✅ Easy tasks → always solved → 0 gradient ❌ Hard tasks → always fail → 0 gradient 💡 Our idea: treat exploration as a knapsack problem → allocate rollouts where they matter most. ✨ Results: 🔼 +20–40% more non-zero gradients 🧮 Up to 93 rollouts for hard tasks (w/o extra compute) 📈 +2–4 avg points, +9 peak gains on math benchmarks 💰 ~2× cheaper than uniform allocation 📄 Paper: huggingface.co/papers/2509.2…

🎉 I'll be attending #ICLR2025 and the #Alignment Workshop in Singapore next week! If you're interested in LLMs, RL theory & algorithms, reasoning, optimization, and multimodal LLMs—let's connect! 🚀 I'll be presenting our latest research: 📌 Preserving Diversity in Supervised Fine-tuning of Large Language Models 📍 Hall 3 + Hall 2B #241 🗓 Fri, Apr 25, 3:00–5:30 p.m. 📌 Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and Intervention 📍 Hall 3 + Hall 2B #231 🗓 Thu, Apr 24, 10:00 a.m.–12:30 p.m. 📌 Adam-mini: Use Fewer Learning Rates to Gain More 📍 Hall 3 + Hall 2B #280 🗓 Fri, Apr 25, 10:00 a.m.–12:30 p.m. Looking forward to insightful discussions and new collaborations—feel free to DM to meet up! ✨ #ICLR2025 #Singapore #LLM

🌟 Can better cold start strategies improve RL training for LLMs? 🤖 I’ve written a blog that delves into the challenges of fine-tuning LLMs during the cold-start phase and how the strategies applied there can significantly impact RL performance in complex reasoning tasks that requires test-time scaling strategies. 👉 tangible-polo-203.notion.sit… 💻 Check out the code here: github.com/liziniu/cold_star… 🧠 Key insights: 1. Diversity's Impact on RL Training: Sampling diverse outputs is essential for LLMs to effectively explore optimal responses during online RL training. 2. Understanding Exploration in LLMs: Unlike classical RL, exploration in LLMs relies on sampling from the knowledge encoded during pre-training. This makes the diversity reduction issue in the cold start phase even more critical for long-term performance. 3. New Approach for Preserving Diversity: We exaime the effectiveness of GEM, a new game-theoretic approach that preserves diversity compared with the commonly used CE methods in SFT. A key innovation is the introduction of a meta-controller to regulate the learning process within the game framework. 🔍 Findings: By maintaining diverse behaviors in the cold-start stage, RL models show improvements in its training. #LLM #Diversity #RL #SFT #Reasoning #Test-time scaling

🚀 Excited to share our paper "Why Transformers Need Adam: A Hessian Perspective", which is accepted at #NeurIPS2024! We delve into why SGD lags behind Adam, uncovering the "block heterogeneity" in Transformers' Hessian spectra that Adam navigates more adeptly. 🧠💡 Our findings show that Adam's coordinate-wise rates are crucial for handling this heterogeneity, a game-changer for optimizing Transformers! Paper Link: arxiv.org/abs/2402.16788