@MSFTResearch | PhD @UWMadison | BS @PKU1898
Joined April 2017
- Tweets113
- Following340
- Followers1K
- Likes618
Pinned Tweet
๐New paper on Looped Transformers!
Latent reasoning is fast, but struggles to match CoT-level accuracy at scale. Can looped Transformers give us both?
We find: yes! A looped padded backbone turns out to be a surprisingly simple recipe that works -- It gives latent thoughts a parallel workspace that can be supervised similarly to explicit CoT. ๐งต
Ying Fan retweeted
๐ ๐๐ก๐๐ญ ๐ข๐ ๐ฅ๐จ๐จ๐ฉ๐๐ ๐๐๐ฌ ๐๐จ๐ฎ๐ฅ๐ ๐ฎ๐ฌ๐ ๐ร ๐๐๐ฐ๐๐ซ ๐ฉ๐๐ซ๐๐ฆ๐๐ญ๐๐ซ๐ฌ ๐๐ง๐ ๐ ๐ร ๐ฌ๐ฆ๐๐ฅ๐ฅ๐๐ซ ๐๐ ๐๐๐๐ก๐ ๐๐ง๐ ๐ฌ๐ญ๐ข๐ฅ๐ฅ ๐๐ ๐๐จ๐ฆ๐ฉ๐๐ซ๐๐๐ฅ๐ ๐ญ๐จ ๐ ๐ฌ๐ญ๐๐ง๐๐๐ซ๐ ๐๐ซ๐๐ง๐ฌ๐๐จ๐ซ๐ฆ๐๐ซ?
Introducing ๐๐๐ญ๐ก๐ข๐ง๐ค๐ข๐ง๐ ๐๐ญ ๐
๐ข๐ฑ๐๐ ๐๐จ๐ข๐ง๐ญ๐ฌ. We argue that looped LMs don't have to pay for every loop. They can scale by adding FLOPs at constant memory.
The key is the fixed point of the loop. We introduce:
โ ๐
๐ข๐ฑ๐๐-๐๐จ๐ข๐ง๐ญ ๐๐ก๐จ๐ซ๐ญ๐๐ฎ๐ญ๐ฌ: what fixed points buy a looped model in training (pre-training, post-training) and inference (prefill, decoding)
โ A ๐ฅ๐๐๐ซ๐ง๐๐ ๐๐๐ฉ๐ญ๐ก ๐ฉ๐ซ๐ข๐จ๐ซ and ๐จ๐ซ๐ญ๐ก๐จ๐ ๐จ๐ง๐๐ฅ ๐ข๐ง๐ฃ๐๐๐ญ๐ข๐จ๐ง: two ways to shape those fixed points
๐งต Long thread ahead (0/n).
Ying Fan retweeted
We've been obsessed with bending the scaling laws recently and our new paper shows that looping with model growth improves the scaling exponent, leading to gain that compounds with compute!
Everyone assumes architectural changes only give constant factor gains and pretraining progress mostly comes from data (e.g. @dwarkesh_sp's recent post).
We found that model growth, looping, and boundary operators result in compute multipliers over standard transformers that grow exponentially with each OOM of compute.
- 1.55x at 1e20 FLOPs and 2.7x projected at 1e25.
- Matches GPT-3 13B on CORE with 20x less compute
w/ @charllechen, @akshayvegesna, @andrewgwils
๐งต
Ying Fan retweeted
Excited to share Ludiโ.โ!
Ludiโ.โ is an agentic humanoid system that integrates real-time dialogue, social reasoning, memory, perception, navigation, and manipulation, while allowing a person to interrupt and redirect the robot during ongoing behavior.
Ying Fan retweeted
I've been unsettled lately. Reading messages or papers feels like dissociating. Everything seems a bit alien, even if it's completely human. I've had a realization: When our simulations finally exited the Uncanny Valley, they brought the Uncanny with them. nsaphra.net/post/uncanny/
Ying Fan retweeted
Earlier today we release our report about a vulnerability that allowed us to read out the encrypted thinking traces from many frontier models (thread below!):
A few thoughts:
First, there is an immediate privacy concern with publicly posted reasoning traces (which is also why we took time to release the report after the initial disclosure). We were able to decode the thinking of many json traces posted online, and found private info in there.
Ironically, during this investigation we also had entry into HF during the cybersec incident due to a leaked prod key (but did not exercise the key beyond a whoami ;)).
Another big update from this for me was actually seeing thinking traces at scale from Anthropic and OpenAI, and all the weirdness and 'casual' misalignment they contain (examples below). I do think model monitoring has an uphill battle ahead of us in the coming year.
---
Finally, there is something to be said for just making thinking traces accessible to all users. I do believe that we would achieve a much broader, much more pluralistic form of oversight through broadly accessible thinking, that would make model deployment safer.
Ying Fan retweeted
teapawt.net/kangwook_lee/status/20โฆ
1. No wonder why this is being liked everyday
2. The exact same technique has been around for "encrypted prompts"
Ying Fan retweeted
Legitimately spent 7 days non stop on this. I don't think I'll work on another theory thing for a while. Verification is an insane bottleneck.
Ying Fan retweeted
1/ Sharing a new, interesting project I did during my internship at Microsoft AI Frontiers w/ @JohnCLangford
TL;DR: At decoding time, we feed **previous hidden state** into the input together with token embedding, and it boosts performance for free.
arxiv.org/abs/2608.08888
Ying Fan retweeted
Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub!
This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people.
Thanks also to everyone who shared ideas and opened PRs with improvements!
Of course, I plan to keep adding new material, including new attention variants and architectures (while bigger projects like RL and Reasoning From Scratch live in their separate repositories).
I am also currently working on a larger applied custom โsmallโ LLM project. It has been keeping me super busy this month, but I will share more on that soon.
If you are new to it, some of the highlights include
1. Of course, the complete code path from tokenization and attention to pretraining, classification, and instruction fine-tuning, etc. All of it FROM SCRATCH, of course! (RL lives in a companion repo.)
2. From-scratch implementations of Llama, Qwen, Gemma, and Olmo (smaller variants that run locally and can be plugged into the training scripts).
3. From-scratch implementations of attention alternatives and other architecture components, such as GQA, MLA, sliding-window attention, Gated DeltaNet, DeepSeek Sparse Attention, cross-layer KV sharing, and mixture-of-experts
4. Materials on KV caching, training performance, memory-efficient weight loading, DPO, evaluation, and LoRA
So, if you donโt have any weekend plans yet, happy tinkering!
Ying Fan retweeted
arxiv.org/pdf/2301.13362
fyi - @yingfan_bot did e2e RL for diffusion many years ago
[2/N] How do existing generative models solve this problem?
They break down generation into many small steps, so each step has essentially one right output and there is nothing to average.
Autoregressive LLMs predict one token at a time. Diffusion denoises a little at a time.
But this means models are trained on one step, then run for hundreds at inference, so training and inference become different. Each step's errors feed into the next, drifting the model onto inputs it never saw in training (exposure bias), compounding errors.
Ever since AlexNet, end-to-end networks have replaced hand-built pipelines across deep learningโclassification, detection, segmentation. Generative modeling is the last holdout, and breaking down generation is exactly what's blocking it.
Ying Fan retweeted
when on-policy enter pretraining be like:
We discovered a third pretraining axis beyond parameters and data: exploration.
Scaling exploration monotonically improves existing models across images/video/language, and unlocks end-to-end generation.
In the simplest case, it's just a for loop.
Introducing Explorative Modeling.
TLDR:
- Gains from exploration grow with scale: 7%โ36% as data scales, 13%โ23% as parameters scale, and gains double at 3ร the compute
- Adding exploration to ~SOTA baselines improves data efficiency by 6.2ร, FLOP efficiency by 4.1ร, parameter efficiency by 47%, and hits a near-SOTA 1.43 unguided FID on ImageNet
- Exploration lets you trade training compute for generalization, and scales how end-to-end your generative model is
- End-to-end Explorative Models (XMs) match diffusion performance on control tasks with up to 256ร less inference compute
๐งตThread:
Ying Fan retweeted
๐ฅ
๐๐๐ฐ ๐๐ฅ๐จ๐ : ๐๐จ๐ฐ๐๐ซ๐๐ฌ ๐๐จ๐จ๐ฉ๐๐ ๐๐จ๐๐๐ฅ๐ฌ ๐๐จ๐ง๐ ๐๐ข๐ ๐ก๐ญ โ ๐๐๐ซ๐ญ ๐
Looped models reuse the same weights across depth, promising a better computeโparameter trade-off, especially for reasoning.
๐๐ฎ๐ญ ๐) ๐๐จ ๐ญ๐ก๐ ๐ ๐๐ข๐ง๐ฌ ๐ฌ๐ฎ๐ซ๐ฏ๐ข๐ฏ๐ ๐ฐ๐ก๐๐ง ๐๐จ๐ญ๐ก ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ง๐ ๐ข๐ง๐๐๐ซ๐๐ง๐๐ ๐
๐๐๐๐ฌ ๐๐ซ๐ ๐ฆ๐๐ญ๐๐ก๐๐? ๐) ๐๐ง๐ ๐ฐ๐ก๐ข๐๐ก ๐๐ซ๐๐ก๐ข๐ญ๐๐๐ญ๐ฎ๐ซ๐๐ฅ ๐๐ก๐จ๐ข๐๐๐ฌ ๐๐๐ญ๐ฎ๐๐ฅ๐ฅ๐ฒ ๐ฆ๐๐ญ๐ญ๐๐ซ?
We run ๐๐ฉ๐ฉ๐ฅ๐๐ฌ-๐ญ๐จ-๐๐ฉ๐ฉ๐ฅ๐๐ฌ ablations spanning Ouro to Huginn. Huginn performs better overall, with the largest gains coming from the loop-in-the-middle (sandwich) design and input injection, though they provide different benefits.
Trained on ๐๐๐๐ tokens, an ๐๐-๐๐.๐๐ Huginn MoE approaches or surpasses a ๐๐๐-๐๐.๐๐ feedforward MoE on several reasoning benchmarks, including GSM8K (83.6% vs. 80.8%), while using ๐๐% ๐๐๐ฐ๐๐ซ resident parameters under ๐ฆ๐๐ญ๐๐ก๐๐ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ง๐ ๐ข๐ง๐๐๐ซ๐๐ง๐๐ FLOPs.
More details and the blog link in the thread โ
Ying Fan retweeted
The answer is deceptively simple imo.
Thereโs a notion of a general-purpose harness (like, say, CoT reasoning, proper compaction, or an RLM) thatโs essentially just what *the* model should become. A better โmodelโ in the API will eventually wrap it and train accordingly.
And a separate notion of a harness thatโs last-mile specification over the underlying bland model. This is irreducible to the extent that your task/context diverges from the mode, regardless of capability. Here, the closer your complexity to the minimum number of bits required, the better.
Ying Fan retweeted
With the recent surge in research on Looped Transformers, we have compiled a list of popular papers tracing the architectureโs development from its earliest forms to its latest iterations.
You can check them out here:
alphaxiv.org/shared/folder/0โฆ
Ying Fan retweeted
Huge congratulations to PKU alumni Wang Hong and Deng Yu, recipients of the 2026 Fields Medal โ mathematicsโ highest honor! ๐งฎ
Announced at the 2026 International Congress of Mathematicians (ICM) in Philadelphia, the pair are among four mathematicians worldwide to claim this prestigious prize, awarded every four years to outstanding scholars under 40.
The awards are historic on several levels.
โFirst time two mathematicians who completed undergraduate studies in mainland China win the Fields Medal in the same year.
โBoth joined Peking University as undergraduates in 2007.
โWang Hong becomes only the third female Fields Medalist ever, the first Chinese woman, and the first recipient with a bachelorโs degree from a mainland Chinese university.
We are immensely proud of their extraordinary breakthroughs and lasting contributions to global mathematics.
Ying Fan retweeted
"Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers"
Latent reasoning lets models think without writing every reasoning step, but it usually loses accuracy as models get bigger.
This paper, LOTUS, fixes this by using a looped Transformer that updates many hidden thoughts in parallel, then trains them to match real CoT steps.
So the model gets CoT-like reasoning without generating the full CoT.
At 3B, it nearly matches explicit CoT on GSM8K while thinking 2.5x faster, and 6.9x faster for long reasoning.
The hidden thoughts can also be read back as reasoning steps, making them compressed but not completely ambiguous.
"Looped / Recursive LMs can solve generalization." Again!
Reminds me of our previous work @Kangwook_Lee: teapawt.net/Kangwook_Lee/status/18โฆ which makes the model itself looped, vs this "loops" the harness around it
Transformers struggle to generalize to tasks they were not explicitly trained on. Instead, we propose in 2026 that it is the job of the harness to generalize through composition.
We observe a powerful property when training RLMs: for tasks with shared structure that look different, the root model naturally learns the same trajectory, meaning it views the two task trajectories as the same! In other words, the Transformer does not need additional generalization capabilities to transfer capabilities from one task to the other, the harness induces it.
We find that well-designed harnesses form a quotient set over task trajectories, meaning their individual LLM calls can see structurally โsimilarโ tasks as near-identical, token-for-token! Harnesses can effectively generalize for the Transformer during training, without relying on any intrinsic generalization capability from the model.
For example, RLMs can see problems of different lengths as the same: we show that RLMs can train exclusively on short tasks, and fully generalize to similar but unseen tasks 8-32x longer because it produces near identical trajectories for both.
Taking this further, we show that tasks across different domains (e.g. math solutions vs. essay writing) that share a decomposition strategy exhibit the same generalization effect. RLMs can train on the problem of finding which essays belong to the same author and improve performance on finding math problems that share similar solutions.
The full blogpost, experiments, and discussion are in the thread below.
