Assistant Professor @ Tsinghua University

Joined June 2016
Kaifeng Lyu retweeted
From LR scaling laws to data curricula, we've been exploring better recipes for LLM pretraining. Many of these ideas have made their way into Puro-2B and Kaiyuan-2B, our fully open, efficient, low-cost pretraining projects. More to explore, and more to come!
1
4
24
1,767
Wrapping up my first month at @DiscoLoopAI (am on leave from @PrincetonCS and @PrincetonPLI ). Excited to work on using AI to accelerate scientific discovery and solve important problems. My first startup experience, and with an incredible set of colleagues!
Announcing Discovery Loop! I am very excited to announce that, along with my longtime friends and collaborators @Sanjay_Ghemawat, @OriolVinyalsML and @quocleix, we are founding Discovery Loop (@DiscoLoopAI), a Public Benefit Corporation whose mission is to automate machine learning, science, and engineering to accelerate discoveries and progress. The four of us have worked together for 14 to 30 years, and have helped build some of the world’s most used products, infrastructure and AI models, and we’re excited to turn our attention to this ambitious endeavor. ♾ Learn more at: discoveryloop.com
32
41
2
857
62,172
Kaifeng Lyu retweeted
Can LLMs generate novel research ideas? We propose Priced Guidance to quantify this via compression! We lower bound P(LLM forecasts a future paper’s idea X). Instead of sampling ideas until X is matched, the LLM plays “20 Questions” to recover X, charging each hint by surprisal.
21
86
22
541
80,707
Kaifeng Lyu retweeted
unfortunately i contacted them on the night of wednesday, september 2nd making the following points 1. i had intel they had information about math work i was doing and a few days before they had spun up a group to try to compete. 2. i emphasized and reemphasized mine was a strictly personal mathematical collaboration 3. that this was a collaboration that had been going on for a year 4. that in my personal collaboration we were happy users of codex as well as obviously claude (and i later complimented astra explicitly after it came out for its help, and reassured i would obviously be sprinkling acknowledgments for it, previous models, and codex liberally in the writeups) 5. that it would look terrible if openai were competing against mathematician consumers, and my collaborator (Tristan) wanted to email about this explicitly 6. that i expected none but in the worst case intended to refuse marketing around the result we were working towards, being the fruit of a personal external academic collaboration, and wanted to express this because i figured worries about such marketing by their leadership was why a team was spun up to extraordinarily directly compete. sebastien and i have in the past months had our fun pushing each other on twitter, but as i process it all i keep feeling bad about Tristan. 1. neither of us could sleep Sunday night, so we went on a walk around the city. i finally asked him why fluids, and he told me it was the navier-stokes problem. he told me parts of the story about his classic nonuniqueness result with Vicol i hadn’t heard yet, and also about how he too grew up writing shaders as a kid and then switched to math 2. tristan is a mathematician’s mathematician. he kept rock solid to his principles during the tense calls and negotiations, immediately turning down a career dream and million dollar prize for me, the sort of strange excited unclear-quasiuseful confused idiot in our collaboration. fifteen minutes later, his feet were up on his desk with three toes per foot peeking out of ripped socks 3. our periodic call after i’d texted him euler blowup was just laughing for an hour or two. just laughing. i’d seen fable deal correctly with monge ampere in the equality case of ehrhart’s conjecture, so in some call before that i’d really pushed him for the craziest shit he could think of that i might try. i texted him the pdf as an update on that “diego luis 3” line, in which following his incredible expression of intuition i hodgepodge slapped together a bunch of our recent work, in particular correcting a bunch of better and lesser known papers in the literature, as well as on diego and luis’ ipm paper, and hoped an analogous construction for boussinesq (thus axisym euler) might work out. i managed to wade through enough confusion to insist to the model it stop telling me stupid regularity stuff about kelvin waves, and a rayleigh-taylor ansatz stuck. like i said, just laughing. when i texted him the pdf, the worst thing either of us had seen so far, i didn’t even comment on the euler blowup claim, only explaining why skepticism should remain for boussinesq, because i found it so shocking. i mean what the hell did i know, i must’ve confused myself by misunderstanding one adjective like i was used to with algebraic geometry. im so lucky to have been able to live that 4. he managed all this this entire time with unclear sleep, timestamps continue to amaze me, due to just recently having a kid it is obvious he deserves the same sort of praise he heaped upon luis, perhaps mutatis mutandis:)
Replying to @__alpoge__
nothing at all was locked we were just willing to talk but you didn't talk to us ... sorry but that's just untrue, we were willing to go above and beyond and have as many discussions as you would have liked to reach resolution.
140
549
88
6,240
721,610
Kaifeng Lyu retweeted
Can Looped Transformers still help when FLOPs, parameters, and KV cache are all matched? We introduce SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers. The answer is yes. And the advantage grows with scale. 🧵 1/8 Paper: arxiv.org/abs/2609.01343
9
71
7
452
75,182
Kaifeng Lyu retweeted
We submit MuonH with minus-sqrt LR scheduler (1-\sqrt{t}) to the modded-nanogpt Track 3 benchmark. It improves the current MuonH SOTA by 75 steps—and, surprisingly, nearly doubles the peak learning rate under this schedule. More results are on the way. Stay tuned.
4
12
3
71
15,470
Kaifeng Lyu retweeted
Finally, our paper is on arXiv! 🎉 Huge thanks again to my amazing collaborators Yiran @zhangyir22, Yingfa @yingfachen, and Kaifeng @vfleaking! 🙌 We complete the global-minima analysis of 2D-RoPE and prove that an $\ell_\infty$-constrained global minimizer found by one-layer 2D-RoPE can learn copy with length generalization. More broadly, we hope this sheds light on how positional encodings shape a model’s learning behavior beyond this specific setting. ✨ arXiv: arxiv.org/abs/2607.16072 P.S. Always happy to chat about theory! 🤩
(1/n)If you paste your data into an LLM and ask it to plot it in Python, it first has to copy every number into the code — so can you trust the result, or might it quietly miscopy a few? Our new paper finds frontier LMs sometimes do get this wrong! Models that can solve hard reasoning problems still fail at a much simpler task: copying an input exactly, token by token. Why? RoPE has an inductive bias that works against exact copying. Better option? we propose 2D-RoPE, which views text as 2D instead of 1D.
2
9
24
3,236
Kaifeng Lyu retweeted
To figure out the mechanism of an unfamiliar environment, an agent has to work out what the world is and how it changes. That's what our [schema] does. On ARC-AGI-3 Public, [schema] thinks like a physicist: it grounds the state and discovers the mechanism in one editable program. This induced world program makes action efficient: once it discovers a mechanism, it plans inside the model for free. And we achieve an amazing 99% RHAE. Starting from here, mechanism discovery in much richer environments is what's next. Stay tuned!
Today, we’re introducing [schema]: a harness reaching 99% RHAE with Opus 4.8 + Fable 5 and 95.35% with GPT-5.6 Sol on ARC-AGI-3 Public set. [schema] makes an LLM think like a physicist. 🧵
2
5
15
1,751
Kaifeng Lyu retweeted
An important law that a lot of people don't know in reasoning composition tasks: Power law > Uniform. **Skew** makes things learn faster!
A question on synthetic data generation: If we want a language model to solve k-step arithmetic problems (such as a+b*c-d=?), with operands from 1 to 100, which training distribution should we use? A. Uniform distribution: Sample these k operands uniformly from 1 to 100 B. Power law: randomly shuffle 1-100 and impose an artificial power law. Sample these k operands according to this power law. ⚡Our ICML 2026 (spotlight) paper shows: Option B is better! Surprisingly, the same idea extends far beyond this simple example to many reasoning tasks that require implicit composition of multiple atomic skills, including multi-hop QAs and synthetic GSM problems. 📄Paper: arxiv.org/abs/2604.22951 📝Blog: zixuan-wang-dlt.github.io/po…
3
16
10,627
Kaifeng Lyu retweeted
Every data curation pipeline shares the same instinct: dedup, rebalance, flatten toward uniform. This work shows the opposite for compositional reasoning: uniform sampling creates a symmetric, nearly-flat loss landscape at init, and GD can't find a descent direction (they prove a CSQ lower bound!). Zipf's law in natural language might not be a bug we need to fix. It might be part of why pretraining learns to reason at all.
A question on synthetic data generation: If we want a language model to solve k-step arithmetic problems (such as a+b*c-d=?), with operands from 1 to 100, which training distribution should we use? A. Uniform distribution: Sample these k operands uniformly from 1 to 100 B. Power law: randomly shuffle 1-100 and impose an artificial power law. Sample these k operands according to this power law. ⚡Our ICML 2026 (spotlight) paper shows: Option B is better! Surprisingly, the same idea extends far beyond this simple example to many reasoning tasks that require implicit composition of multiple atomic skills, including multi-hop QAs and synthetic GSM problems. 📄Paper: arxiv.org/abs/2604.22951 📝Blog: zixuan-wang-dlt.github.io/po…
1
1
13
1,457
Kaifeng Lyu retweeted
We’re entering a data-scarce regime; but is less data always worse? Our #ICML work shows that small-set training (e.g., with repetition) can be actively leveraged as a useful optimization bias for structured reasoning tasks. Come chat with Jingwen and Ezra in Seoul to know more!
2
24
2
118
19,451
Kaifeng Lyu retweeted
Excited to share our ICML 2026 paper "SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection"! Our finding might be surprising: for knowledge injection, just 7 carefully designed prompt templates, scaled up to generate synthetic data, can outperform several complex methods across three benchmarks, and stay robust across different generator and target model families. By showing the effectiveness of this simple baseline, we hope it serves as a useful reference point for future work on scalable, robust, and principled synthetic data generation for continual pretraining. 🔈P.S. I'm actively seeking PhD positions starting Fall 2027. Please feel free to reach out or grab a coffee at ICML!!
A bitter lesson for knowledge injection: simple prompt engineering, when scaled to generate large amounts of synthetic data, can beat more complex methods. To inject knowledge into LLMs for data-scarce domains, prior work builds complex methods for synthetic data generation: some are based on RL to optimize the generator, and some involve multi-stage prompting pipelines. In our ICML 2026 paper, we propose SPA (Scaling Prompt-engineered Augmentation), a simple but tough-to-beat baseline: built from just 7 prompt templates, it outperforms several strong methods. 🧵
3
7
1,364
Kaifeng Lyu retweeted
Excited to share our new paper: “Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D.” We show that frontier language models can struggle with a seemingly simple task: copying the input exactly. Viewing text in 2D can help models learn copying better.
(1/n)If you paste your data into an LLM and ask it to plot it in Python, it first has to copy every number into the code — so can you trust the result, or might it quietly miscopy a few? Our new paper finds frontier LMs sometimes do get this wrong! Models that can solve hard reasoning problems still fail at a much simpler task: copying an input exactly, token by token. Why? RoPE has an inductive bias that works against exact copying. Better option? we propose 2D-RoPE, which views text as 2D instead of 1D.
3
2
7
992
I am starting a blog about deep learning theory and its value to practitioners! First post is about Adam, broken convergence proofs, and what theory can contribute when stuff just works anyways without it. Subscribe on Substack if you like it! undertheassumptions.substack…
🤖 Made with AI
6
86
11
438
108,811
Kaifeng Lyu retweeted
Playing with an optimizer speedrun is something that never gets old. Built on top of tinyurl.com/muonh and Claude should take all the credit for hypertl tuning.
New modded-NanoGPT optimization benchmark result: @wen_kaiyue has improved upon both the Muon and AdamW baselines, by replacing their weight decay with hyperball optimization. The new record is 3325 steps.
3
16
132
18,940
Kaifeng Lyu retweeted
I won't be at ICLR this year but @xingyudang will help present Fantastic Optimizers arxiv.org/abs/2509.02046! Stop by at Pavilion 4 P4 5309 this afternoon to see what we have found in extensive sweeping and more importantly, what we learned after the paper that leads to Hyperball!
12
1
108
12,457
Kaifeng Lyu retweeted
📢 Come meet us at #NeurIPS2025! We'll be presenting our paper: Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold 🗓️Friday 5 Dec 4:30 p.m. PST-7:30 p.m. PST 👀Exhibit Hall C,D,E Poster Location: #5102 Expect your feedback!
Adam prefers a different minimizer than SGD (exemplified below), but how? 🤔 Our NeurIPS 2025 Paper: Based on our Slow SDE approximation of Adam, we show that under label noise Adam implicitly minimizes tr(Diag(H)^½), whereas prior works showed that SGD minimizes tr(H). 🧵1/n
1
1
8
4,264
Kaifeng Lyu retweeted
Excited to share our new method ✏️PENCIL! It decouples space complexity from time complexity in LLM reasoning, by allowing model to recursively erase and generate thoughts. Joint work w. my student @chenxiao_yang_ , along with @BartomNati and @McAllesterDavid.
I've discovered a truly marvelous idea for building AGI, but Twitter's space limit won't let me explain it! Damn! 😫 Introducing ✏️PENCIL, a new LLM reasoning paradigm that generates and erases thoughts, enabling longer and deeper thinking with shorter context. #ICML2025 🧵1/n 🤯 Theoretically, ✏️PENCIL is Turing-complete with optimal space and time complexity, and thus can solve arbitrary computable problems efficiently. This is something fundamentally impossible for ✒️CoT. This post is based on the paper “PENCIL: Long thoughts with short memory” accepted in ICML 2025, a joint work with Nathan Srebro, David McAllester, and @zhiyuanli_ Paper: arxiv.org/pdf/2503.14337 Github: github.com/chr26195/PENCIL Expand thread for full details ⬇️
1
10
35
6,174
We will present this paper at #ICLR2025! 1. 𝐎𝐫𝐚𝐥 𝐒𝐞𝐬𝐬𝐢𝐨𝐧 𝟏𝐃 (𝐓𝐡𝐮𝐫𝐬𝐝𝐚𝐲 𝟏𝟎:𝟒𝟐𝐚𝐦) @PandaAshwinee will give a talk 2. 𝐏𝐨𝐬𝐭𝐞𝐫 𝐒𝐞𝐬𝐬𝐢𝐨𝐧 𝟒 (𝐅𝐫𝐢𝐝𝐚𝐲 𝟑𝐩𝐦) Come to chat with @PandaAshwinee @vfleaking @infoxiao @abeirami Unfortunately, I won’t be at Singapore in person. But my DM is always open :)
Our recent paper shows: 1. Crrent LLM safety alignment is only a few tokens deep. 2. Deepening the safety alignment can make it more robust against multiple jailbreak attacks. 3. Protecting initial token positions can make the alignment more robust against fine-tuning attacks.
1
8
56
6,257