Ruoyu Sun

We’re excited to share our work "A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning". An earlier version of this work has been on arXiv for a few months. We added more experiments and revised it to this new title. The recipe is simple: the model samples its own reponses at low temperature, learns from them with ordinary SFT training, and repeats. No reward. No verifier. No fancy objective beyond standard SFT. On Qwen2.5-Math-7B, mean Pass@1 over 6 math benchmarks improves 22.7 → 39.5. Note that mean Pass@32 also improves 61.0 → 67.9, suggesting that this simple reward-free procedure unlocks more of the model’s existing reasoning potential. See the updated paper directly at: github.com/ElementQi/SePT/bl… The arXiv link is: arxiv.org/abs/2510.18814 The updated version will appear on arXiv shortly. @Phanron_xli

OpenAI's list is here. And so is Nobel week. I read about neutrinos—how they pass through the universe untouched, changed by nothing. I was moved. I read about left- and right-handed molecules. I always use chirality when teaching determinants in linear algebra, but when students ask why life is single-handed, I have no answer. I'm glad people have studied this for a long time, and the solution of chemists has an interestingly algorithmic flavor. The matrix multiplication story is moving too. From Strassen's algorithm to decades of grueling progress. I want to know if 2 is the limit. It just dropped below 2.25, but I don't see much discussion. What about the Riemann Hypothesis? 3SUM? UGC? I hope someone tells their stories. If you know a good math story, tell it. The world is listening. #NobelPrize #Math #SciComm

Cannot wait to see how math starts to help solve problems in other complex domains.

There are different types of math: --proving conjectures using exisiting tools; --creating brilliant tools for proving conjectures; --creating new math frameworks for math; --formulating math problems for an applied area; ----creating new math framework or theory for an influential new applied area (e.g. Turing, Shannon, Kalman). I’m particularly interested in AI producing math theories for applied domains. Unfortuantely, the last two tasks can be painfully difficult, especially the last one. “Within 2 years” may be enough for proving some math results in applied domain journals, but not enough for making a real impact. In many domains, the bottleneck may be a lack of data, both for training and inference. The input is no longer a half-page problem statement. I would be glad if there is some promising sign in 5 years.

Excited to share our paper on continual learning & reducing forgetting at #ICML2026 tomorrow! 📍 Poster Session 3: HALL A #2010 Wed, Jul 8, 2026 • 9:30–11:15 AM CST Unfortunately, I'm not at ICML; but my students @SenmiaoWang will be happy to chat. Forgetting is a challenge for continual learning, including sft. 1) Finding: Correlation of distance and forgetting. We found that optimizers can converge to minima with similar SFT loss with different distances from the pre-trained model. The farther the model moves, the more pre-trained knowledge it tends to lose. 2) Algorithm: **MoFO**, a Momentum-Filtered Optimizer for LLM fine-tuning. At each step, MoFO updates only the parameters with the largest momentum magnitudes within each block, leaving the rest unchanged. This biases toward closer solutions, preserving more pre-trained knowledge while maintaining strong task performance. No pre-training data required. No extra model-weight storage. Paper: arxiv.org/pdf/2407.20999 [TMLR, J2C Certification]

Glad to give a talk at Oxford this Tuesday at 1pm, as I continue my UK visit. I will use Hessian and spectra — two classical mathematical objects — as entry points into my recent works on LLM training. maths.ox.ac.uk/node/81480 Side: This post was first flagged as automated...how to prove that it is a human-written post?? tried a few times and failed; waited 20 mins to resend, got it posted.

I’m excited to share that I will give a plenary talk on neural network optimization at the SIAM Conference on Optimization (OP26) in Edinburgh, United Kingdom, at 8am, June 3. I'll describe my view of the optimizaiton foundation needed for neural net training, from both the global landscape and local structure perspectives. Check out the conference program and more information here: siam.org/conferences-events/…. Hope to see you there! #SIAMOP26

ICML Reviewer kindly suggested we cite a paper posted on arXiv in March 2026 as related work. For reference, the submission deadline of ICML was January 28, 2026. A fair request, assuming authors are now expected to survey not just the literature, but also the future.

With the gemini3 hype, I tried my own test math problem, but... it fails. Prompt: "Randomly pick 3 points in an n-dim sphere. What is the probability that the projection of the center to the plan spanned by the three points is inside the traingle formed by the three points?" Gemini 3pro thinking : 1/4 for any n (tried two times, each with > 3 mins thinking) deepseek : 1/4 for any n (>7 mins thinking) GPT5.1 thinking: 1/2 for any n (> 5 mins thinking) Guess: which answer is correct? I don't have time to check pass@8 for all of them. To be fair: My testing result is that the pass@8 of GPT5.1 thinking will give the correct answer though.

Glad to see Adam-mini's idea can be adopted to improve muon, to get NorMuon. Our original motivation of Adam-mini is to improve memory, but it turns out Adam-mini's block-wise scaling can be used to improve performance in a few scenarios

Neural nets' Hessians are often **nearly block diagonal**! There is little understanding when and why this will happen. We provide one of the first theoretical analysis using random matrix theory. Somewhat unexpectedly, we find that one primary driver is a large # classes C. For LLMs, this C > 10^5. Check out more details in our paper arxiv.org/pdf/2505.02809.

Two bits to share after watching ICLR Test of Time Award speech: 1️⃣ Adam paper was initially rejected at ICLR 2015!! --Authors successfully appealed (how many would have given up?) This reminded me of DFP algorithm, whose 1st version was rejected initially... 2️⃣ Honored to be cited by the authors in the speech: our NeurIPS'22 work on Adam's convergence & ICLR'25 work on Adam-mini (see Fig. 2 & 3). What's the optimizer for next decade? Let's work together on it! #ICLR2025 #testoftimeaward

Come to our #ICLR25 posters today on GEM and Adam-mini if interested in LLM training! I will unfortuantely miss the first few days and join workshops. 1) GEM, in paper "Preserving Diversity in Supervised Fine-Tuning of Large Language Models". It shows how to improve diversity of LLM post-training. It also won the NeurIPS'24 LLM fine-tuning workshop best paper runner-up! Time: Fri 25 Apr 3 p.m — p.m Location: Hall 3 + Hall 2B #241. See the blog tangible-polo-203.notion.sit… for an intro. 2) Adam-mini, a lihgtweight version of Adam based on principled understanding of neural network Hessian (I had a few posts about it). Time: Fri 25 Apr 10 a.m. — 12 p.m. Location: Hall 3 + Hall 2B #280.

Glad to see the community discussing the effect of beta1 and beta2 in Adam! Similar to this post, we had a sweep of beta1, beta2 before in our NeurIPS'22 paper arxiv.org/pdf/2208.09632 Main findings: 1) Our paper uncovered a striking phase transition in the (β₁, β₂) plane – error suddenly jumps from small to huge! Simon's post reminded me of the 'wow' moment we had. 2) Key suggestions on tuning beta1, beta2 by our theory of phase transition: 1️⃣ β₂ has a 'sweet spot' (treat it like LR: big enough but not too big) 2️⃣ Keep β₁ modest 3️⃣ Bonus insight: Smaller batches need larger β₂ (confirmed by Carmon et al.'s new scaling law paper [R2]) References: [R1] Adam Can Converge Without Any Modification On Update Rules. arxiv.org/pdf/2208.09632 @ericzhang0410 [R2] arxiv.org/abs/2406.19146:“ ...we find that tuning (enlarging) the AdamW 𝛽2 is essential at lower batch sizes.”

If you are interested in GRPO by deepseek, you might want to try ReMax, which is quite related to GRPO but perform better than GRPO. （ReMax was released in Oct 2023, and GRPO was released in Feb 2024) Similarity: (see 1st image) --Both are variants of REINFORCE; --Both add a term in the update. Difference: (see 1st image) --ReMax's term is the reward of a greedy policy, which is deterministic. There is some theory on variance reduction for ReMax. --GRPO's term is empirical mean of reward, which is stochastic. Performance: A later work compared two methods on Llama3.1-8B, and found ReMax is on average about 2 points better than GRPO in a few benchmarks. In addition, ReMax is now available on Verl distributed framework! 🛠️ 🔑 Key Features: - Higher Throughput than PPO - Sibling of GRPO, and perform better - Stable Training with theoretical guarantees on variance reduction Check out the code here: 👉 github.com/volcengine/verl/t… Paper (ICML'2024) arxiv.org/abs/2310.10505 @ZiniuLi @ericzhang0410 #LLM #RL #Reasoning #ReMax #EfficientTraining

Will be at finetuning workshop, an oral presentation of a diversity keeping sft method GEM. It is a game formulation, with one player using RpGAN loss and another play using entropy regularized loss. Improving code pass rate by 7 points on Llama3.1-8B @ZiniuLi @chcoli

#NeurIPS2024 I will present "Why Transformers Need Adam: A Hessian Perspective". East Exhibit Hall A-C #4803. Thursday 11am-2pm. In short: Transformer has Hetergeneous blocks of Hessian, making SGD bad. Will also be at FTML (fine-tuning) workshop at Saturday.

Heading to Vancouver for #NeurIPS2024 and stay all week. Will be happy to chat about LLM optimization, LLM theory, LLM applications, etc. Don’t hesitate to reach out (DM or whova)!

Excited to share our paper "Why Transformers Need Adam: A Hessian Perspective" accepted at @NeurIPSConf Intriguing question: Adam significantly outpeforms SGD on Transformers, including LLM training (Fig 1). Why? Our explanation: 1) Transformer’s block-Hessians are heterogeneous in terms of spectrum (Fig 2); 2) If block Hessian are heterogeneous, then Adam performs better than SGD. --Universal observation on not just GPT, but all architectures we tested, such as ViT, BERT, pure MLP, MLP-mixer, even quadratic problems. Metric of Hetergeniety: --Average JS distance of block spectrum and common optimizers for them: ResNet18: 0.1 [SGD] VGG16: 0.09 [SGD] MLPMixer: 34.9 [Adam] Bert: 53.4 [Adam] GPT2: 83.2 [Adam] ViT-base: 286.4 [Adam] --Calculate the Hessian-Hetergeniety index of your own architecture, using code at github.com/zyushun/hessian-s… Preliminary theory: On block-diagonal quadratic functions, we show Adam's convergence speed depends on block-wise condition number. This condition number turns out to be much smaller than regular condition number, on Heterogeneous quadratic problem we tested. BTW: This paper motivates Adam-mini, which goes deeper and designs a new algorithm. Paper Link: arxiv.org/abs/2402.16788 #NeurIPS2024

Now LLaMa Factory incorporates Adam-mini! 27% total memory reduction when fine-tuning QWen-1.5B. To run it immediately: github.com/hiyouga/LLaMA-Fac…