Ph.D. Candidate @Mila_Quebec | Visiting Researcher @AIatMeta | Former: @MSFTResearch @IITKanpur
Montreal
Joined August 2016
- Tweets427
- Following717
- Followers833
- Likes1.3K
Pinned Tweet
[1/9] While pretraining data might be hitting a wall, novel methods for modeling it are just getting started!
We introduce future summary prediction (FSP), where the model predicts future sequence embeddings to reduce teacher forcing & shortcut learning.
📌Predict a learned embedding of the future sequence, not the tokens themselves
Divyat Mahajan retweeted
What is science, really, and why do we do it?
In my new post I discuss the plurality of perspectives on these questions. As we automate the scientific process with AI, it is important to hold onto the deeply human origins of this collective practice!
mj10.github.io/blog/2026/per…
Divyat Mahajan retweeted
Very excited about this work! Also a good time to share that I joined @GoogleDeepMind earlier this year in @egrefen 's team. Its been a great ride!
Very excited by the bleeding edge modelling that went into this thanks to new learning methods produced by my team, in collab with Gemini Robotics.
Congrats Michelle Lo, @HolarisSun, @MartinKlissarov, @JonnyCoook, @silviasapora, @Aniket_d98, and Claudia Clopath on the launch!
Divyat Mahajan retweeted
We've posted a 156-page tech report on Beam! We have good systems and it's worth reading. It covers all phases through pre-training, mid-training, SFT, RL, and OPD and goes into the infra, data, and scientific decisions we made throughout the process
📚 reflection.ai/documents/beam…
We've released the weights for Beam!!
🤗 huggingface.co/reflection
📄 Model card: reflection.ai/beam-model-car…
🧪 Try Beam in our playground & get an API key: platform.reflection.ai/playg…
📚 AND we take open model safety seriously and have posted our safety report: reflection.ai/beam-safety-re…
Divyat Mahajan retweeted
We can't improve what we can't measure, and LM safety and capabilities are limited by a lack of scaling laws for advanced abilities.
Scaling up past 'emergence' can work, but can be wildly costly. Instead of scaling up the model, what if we could scale down the evals?
Can we forecast benchmark accuracy from models that score zero?
We (co-leads @LukeBailey181 @rckpudi) turn a benchmark into a series of easier hinted versions. Jointly modeling scale C and hint level H enables forecasting GSM8K acc (H=0) within 3 pts from 5600x less compute 🧵
Divyat Mahajan retweeted
I am quite excited about this work on scaling gradient based data attribution to find innocuous looking harmful data.
"What's in your Safe Data" (@LuxiHeLucy) was one of my favorite papers from 2024, and Emergent Misalignment (@BetleyJan) has been one of the most important papers from 2025. Scalable gradients can connect the two. A few things that are exciting here:
1. Gradient attribution can scale: We built on top of the LoRIF paper (Li et. al.), and further simplified it. Gradients are now only 3x slower than standard embeddings.
2. Small models can be strong monitors. A 0.6B model can be used as monitor for harmful data that may enter the training of much larger models.
3. EM data might only detectable after training: The representations in gradient space often align after the model itself has had exposure to the safe-looking harmful data. This is quite intriguing, as it suggests attribution rather than filters as a path towards preventing EM.
teapawt.net/datologyai/statu…
1/7 New Labs Research: Follow the Gradient
SafeTulu filters benign-looking data that weakens LLM safety, cutting attack success up to 22% relative while largely preserving capability.
🤗 Space: huggingface.co/spaces/Datolo…
Data: huggingface.co/datasets/Dato…
Paper: datologyai.com/blog/follow-t…
Divyat Mahajan retweeted
🐝 How to Train a Swarm
In our first blog post, we explain how latency pressure can incentivise parallel work, the communication primitives agents need to co-ordinate, and outstanding challenges in training large agent collectives.
gr.inc/blog/how-to-train-a-s…
Divyat Mahajan retweeted
Yesterday, I left Microsoft which of course feels rather bittersweet after 14.5 years. Nevertheless, the logic that it's the right decision for me seems inescapable.
Divyat Mahajan retweeted
Multi-Token Prediction (MTP) can be done much better.
🚀 Turn out,you can recover the gradient for KL self-distillation to next-token prediction (NTP) for free without ever computing the KL. You just need a one-line code change:
h_ntp += h_mtp.detach() - h_mtp
Soft Multi-Token Prediction is pretty much a free lunch for MTP.
📜 Paper: arxiv.org/abs/2610.38331
💻 Code: github.com/JaydenTeoh/Soft-M…
🤗 HuggingFace: huggingface.co/papers/2610.3…
ALT Soft Multi-Token Prediction Illustration
Divyat Mahajan retweeted
Couldn't be happier to announce my new paper on defining compositional structure — and generating it! I see this project as the culmination of my PhD, as it unifies all my interests: compositionality, compression, structure, and open-endedness. More below! arxiv.org/abs/2610.20922
Divyat Mahajan retweeted
Decided to revive the blog and write a short post about embracing the agents and increasing the scope of what kind of research is possible. Very much in line with the general experience but a good excuse to get back to writing blog posts (more soon!)
mj10.github.io/blog/2026/fem…
Divyat Mahajan retweeted
Imagine giving Claude Opus 5 ten questions it reliably answers individually. Put them in one prompt, and it starts getting some wrong.
Excited to share our new work that investigates why. Our findings point to “implicit budgeting” in LLMs: as more questions share a prompt, reasoning per question shrinks, following an approximate power law (even with plenty of context space left). Thus, questions that need longer reasoning are hit hardest.
Understanding this also helps us batch questions more efficiently. For Opus 5 on GPQA-Diamond, that saves ~24% of tokens with roughly unchanged accuracy.
Huge credit to @chenpan_li for the thoughtful experiments and hard work behind this paper!
Divyat Mahajan retweeted
Excited to share first glimpses of our research on agent collectives, a year in the making.
I’ve been thinking about parallel reasoning for some time since I was at Meta. Why automate researchers when we should be automating communities? Genius is great, but scenius matters too.
We’re sharing a recipe today (RLCR) that is producing interesting results for us at a small scale. We’re using it to train our Avalon-M1 series of models, starting with a small 120B model.
Usual caveats apply. We are a small team (8 people!), we are conducting more experiments and ablations as we speak, and we’re still training and seeing gains.
Let us know your feedback. My summary of findings in thread below. 🧵
Introducing Reinforcement Learning from Collective Rewards (RLCR).
RLCR is a recipe that trains language models as agent organisations, which share a message board and workspace. We use it to train Avalon-M1-Mini (120B) and study its behaviour.
We find a number of interesting results, including:
🛠️ Learnt organisational hierarchies and divisions of labour
🕵️ Increased collective verification, through hires of verifiers and testers
👥 The rise of collective thinking, where agents think more about other agents and how to help them
We believe RLCR can provide a laboratory for studying collective intelligence and multi-agent safety.
Blog post and technical report 👇
Divyat Mahajan retweeted
Introducing Reinforcement Learning from Collective Rewards (RLCR).
RLCR is a recipe that trains language models as agent organisations, which share a message board and workspace. We use it to train Avalon-M1-Mini (120B) and study its behaviour.
We find a number of interesting results, including:
🛠️ Learnt organisational hierarchies and divisions of labour
🕵️ Increased collective verification, through hires of verifiers and testers
👥 The rise of collective thinking, where agents think more about other agents and how to help them
We believe RLCR can provide a laboratory for studying collective intelligence and multi-agent safety.
Blog post and technical report 👇
Divyat Mahajan retweeted
Check out @hazel_heejeong latest paper. This was a pleasure to work on. Turns out now that we know World Models can be disentangled we can move on to Compositional Generalization: Has your model actually learned the dynamics of the world or is it just predicting a biased policy?
Your world model may know less about the world than you think.
I've always wondered what world models really learn.
Humans don't memorize every trajectory. We learn rules we can reuse.
Can world models do the same?
We built AnyGoal to find out.
hazel-heejeong-nam.github.io…
Exciting work led by @hazel_heejeong on benchmarks for evaluating compositional generalization in world models!
Your world model may know less about the world than you think.
I've always wondered what world models really learn.
Humans don't memorize every trajectory. We learn rules we can reuse.
Can world models do the same?
We built AnyGoal to find out.
hazel-heejeong-nam.github.io…
6
460
Divyat Mahajan retweeted
Spent years watching models ace a single image and completely whiff on anything that moves or changes angle. Today we’re fixing that.
Try Glimpse now, our first visual thinking model with competitive frontier performance compared to Astra on key public visual reasoning benchmarks.
Divyat Mahajan retweeted
🚨 AutoformBench: A Lean Formalization Benchmark for Advanced Mathematics
Our team at @AIatMeta is excited to release AutoformBench, a new formalization benchmark that pushes the Lean 4 coding capabilities of frontier models to their limits.
While SOTA models have become extremely competent at single-shot problem-solving, formalizing mathematics at scale and cost-effectively remains difficult. For instance, only about 42% of the results in openai/math could be formalized before release.
Furthermore, it is not clear and publicly known how much it costs to formalize big results, which is critical information if formalization is to become a part of the peer review process. This is the focus of AutoformBench.
The benchmark comprises 341 proving tasks drawn from diverse graduate- and research-level mathematical sources and formalized by human experts. Each task requires writing a substantial amount of auxiliary code, including lemmas, definitions, and more, placing the emphasis on models’ ability to build formal infrastructure rather than on mathematical cleverness alone.
The result is a demanding test: even the strongest models require tens of millions of tokens to complete 50% of the tasks (roughly $10 to $100 at current API rates).
Paper (arXiv coming soon): github.com/facebookresearch/…
Code: github.com/facebookresearch/…
Joint work by myself, Ahmad Rammal (@Ahmad3Rammal) and Vivien Cabannes, together with Niket Patel (@niketnpatel), Fabian Gloeckle (@FabianGloeckle), Bartosz Piotrowski, Amaury Hayat (@Amaury_Hayat), Remi Munos (@MunosRemi) and Julia Kempe (@KempeLab) from @AIatMeta, @NYUDataScience, and Ecole des Ponts. (1/6)
Divyat Mahajan retweeted
Parallel attempts repeat work. Long reasoning chains lose detail through compaction. How can long-horizon inference COMPOUND? 📈
GitSwarm: Decentralized Compounding Inference.
A persistent memory swarm that preserves and builds on work in a shared Git repository. 🧵
[1/n]
Divyat Mahajan retweeted
I'll be in SF, attending #COLM2026 from October 6th - 9th, presenting our work on investigating the use of KL estimators in RL post-training of LLMs.
arxiv.org/abs/2512.21852
Come say hi if you'd like to talk about LLM post-training, agentic swarms or automated discovery!
Divyat Mahajan retweeted
Automated AI theorem proving has moved the frontier: The holy grail is not an AI that can produce an endless pile of true theorems.
It’s an AI that can discover mathematics, and build each discovery into the foundation for the next.
Check out our new paper on Learning to Discover *Interesting* Mathematics.
arxiv.org/abs/2609.28603
“An accumulation of facts is no more a science than a heap of stones is a house.” - Poincaré
LLMs are now proving results that have resisted mathematicians for decades. Finding interesting theorems without human guidance is a new bottleneck. We show that we can teach an LLM to do it!
TL;DR:
→ a quantitative notion of interestingness
→ 4.3× higher interestingness
→ a self-expanding discovery loop
[1/5]
