AI Researcher
Milan
Joined October 2021
- Tweets378
- Following142
- Followers1.9K
- Likes651
Pinned Tweet
Ever wondered why language model embeddings are organized semantically or why the weight tying technique is so effective? 🤔
Our new paper, spotlighted at ICML24, dives into these questions.
Joint work with @w_cazzola.
🔗 Read more: openreview.net/forum?id=yyYM…
🧵👇(1/7)
I noticed in the recent Step 5 blog that they used fp32 hidden states to address stability issues. I explored this idea too, but didn’t see any improvements. I only tested small models, though, so this might be a good time to push it again.
f14-bertolotti.github.io/pos…
Super interesting blog! They do not say LR and WD explicitly, but one could reverse them from their laws:
Muon = 165ms = 2.1% of step ->
~7.9s/step ->
~370k steps, ~135M tok/batch ->
LR ≈ 4e-4, WD ≈ 0.1
(just an estimate)
We now have state-of-the-art pre-trained base models in the 3T-parameter weight class, built entirely in-house at MAI.
Post-training results coming soon -- stay tuned!
Lots of details in our technical blog post.
microsoft.ai/news/frontier-p…
Noice
Meet Mistral Large 4, aka Le Chonk.
• 1T parameters, natively multimodal. 49B active.
It is the best open weights model from US or Europe on aggregated benchmarks.
• State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding.
• Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure.
• Available to all via API today. Working with cybersecurity partners privately.
Open weights release end of October.
Francesco Bertolotti retweeted
Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
BF16 rounding in FlashAttention breaks a softmax gradient symmetry, blowing up late-training gradients; the authors fix this using GProj.
arxivb.org/abs/2609.34272
Found a nasty bug in FSDP2 when CPU offloading is enabled. RNG generation mistakenly takes the GPU path, which resets the CPU RNG state and causes the same generation to be produced over and over. TorchTitan is affected too!
github.com/pytorch/pytorch/p…
Awesome read! With tons of insightful experiments and things to look for in a MoE training
🚙New blog on saving experts from an untold, silent failure mode in ultra-sparse MoEs :
While pursuing ultra-sparse MoE training (top-8/768), we found that lower-layer experts can become functionally useless while training/validation loss and load balance remain normal. All experts in these layers have much smaller weight norms than those in healthy layers. We call this silent expert death[Figure 1].
And it wasn’t just us. Auditing public MoEs, we found similar early-layer collapse signatures. For example, MiMo-v2.5-Pro and Qwen3.5-397B-A17B [Figure 2]. Benchmarks can hide unused capacity.
Why? This is not merely a load-balancing failure. We found the learning signal itself disappears. Shallow-expert momentum (the accumulated gradients), as well as the square root of second moment, fell by up to ~3 orders of magnitude while deeper layers stayed stable[Figure 1]. By the time norms collapse, the model has already largely stopped relying on the lower pathway.
We introduce LM Loss as Auxiliary Loss (LLAL): temporarily connect an early MoE layer to the LM head during the first few thouands steps then remove it. On 60B /180B models, this simple strategy rescued the lower layers—and the gain persisted after removal. It improved nearly every aspect of expert learning:
• Better validation loss [Figure3]
• No silent expert collapse [Figure 4]
• Better downstream metrics [Figure 4]
• Markedly better load balance
Check out our blog for more details! A bunch of experiments inside—enjoy the 40-minute read :)
alltoall.notion.site/save-lo…
Struggled with Peak GPU memory with Ring Attention in Transformer Engine?
The issue is that comm buffers grow to hold the full context, even though only ~2× local context is needed at a time.
github.com/NVIDIA/Transforme… fixes this by freeing KV chunks as soon as they arent needed
Very nice read on Hyperball and LR tuning with tons of insights
🐳 New blog finally comes out!
TL;DR: ELR, which controls directional changes in model weights, is a more intrinsic hyperparameter to tune than LR. Directly tuning ELR leads to better schedules that outperform conventional alternatives across scales.
hy.tencent.ai/research/elr?l…
ALT Modded-nanoGPT Track 3 optimization benchmark results. Our MuonH + minus-square-root schedule reaches the 3.28 validation loss target at step 3175, setting a new SOTA for MuonH.
New blog post! When training LLMs we keep every accumulator in full precision to avoid error accumulation: weights, optimizer state, gradient all-reduce. Except one: the residual stream. What if we gave it fp32 too?
f14-bertolotti.github.io/pos…
Interesting alternative to GRPO without verifiable rewards. The core optimization still follows GRPO, but instead of using task-verifiable rewards, it relies on a learned reward metric trained on positive examples (with hints) and negative samples.
🔗 arxiv.org/abs/2607.18470
New paper on pretrain-RL: optimal RL share goes from 20%->30% as model size grows 50M->700M. RL reinforces good top-k moves on easy puzzles, but on hard ones it surfaces (and sometimes reinforces) actions from the distribution tail, both good and bad.
🔗arxiv.org/abs/2607.16097
TorchTitan was also affected if you were using (1) the default dataloader and (2) datasets==5.0.0 which was fairly likely if you manage your project with uv.
As a temporary workaround, TorchTitan now pins datasets to an earlier version.
github.com/pytorch/torchtita…
Found a nasty bug in HF's datasets: after the first restore, the state_dict gets misaligned with the worker's, so a faulty state_dict gets saved. That faulty state then causes the dataloader to restart on the second restore. It should be fixed on main.
github.com/huggingface/datas…
Found a nasty bug in HF's datasets: after the first restore, the state_dict gets misaligned with the worker's, so a faulty state_dict gets saved. That faulty state then causes the dataloader to restart on the second restore. It should be fixed on main.
github.com/huggingface/datas…
In this work, the authors weight a layer update with the previous and following one. As a consequence, close layers make more aligned updates to the stream. They tested this on pretraining, RLVR, and a bit of vision.
🔗arxiv.org/abs/2606.30813
Quite a jump from vanilla OPD. Here the authors use the difference between a privileged teacher and privileged student to compute a token-level advantage. The advantage is used to switch from weak/strong distillation modes.
🔗arxiv.org/abs/2606.30626
From today’s arXiv: the authors investigate how MLP parameters should be allocated across depth. They find that assigning more parameters to earlier layers improves performance, while the reverse allocation hurts it. Cool work!
🔗arxiv.org/abs/2606.23670
This TRM variant makes a transformer block a contractive map, so that looping becomes a fixed-point process. They leverage this by approximating gradients with Neumann series (Truncated BPTT). Very cool work!
🔗arxiv.org/abs/2606.18206
Stellar performance from a 3B model. These results were achieved primarily through post-training refinements on Qwen2.5-Coder. The paper doesn't provide many details, but it appears they distill from RL ckpts and then do a final RL-based instruct RL.
🔗arxiv.org/abs/2606.16140
Interesting router design for MoE. The idea is that an expert’s routing vector should reflect how the expert acts on a token. This work uses the expert’s principal singular direction as the routing vector, plus a few tricks. The results look promising.
🔗arxiv.org/abs/2606.12397
Francesco Bertolotti retweeted
i'm having a really hard time understanding how this can be a good decision
> lying to the user by modifying the weights/prompt sets a very bad precedent and is extremely unaligned
> there is 0 public communication from anthropic about it except a section hidden in a 319 page system card
> it's impossible to know the scope of this safeguard. if you are doing a PR to pytorch does this count? if you are working on kernel development? data collection pipeline for a new eval? this will create a paranoia for every researcher in the field
> you actually don't know how your model is modified, if it's PEFT (modification at the weight level) or steering does this mean your other queries are also biased? is it at the user level or organization level?
there is also the more "moral" argument that the reason why anthropic is able to train this model is ai researchers who will not have access to the model's capabilities anymore. even if you consider that this is the right thing to do, doing it like that is just a lack of respect to the ai research community
in addition to all of that, it's not clear if the safeguard acts on "model autonomy" or "model capabilities" to do ai research. this is very different and my understanding is that it's the latter, and there is almost 0 RESULTS about this in the system card except a vague "2.3.6 Internal measures of AI R&D acceleration" section citing the previous RSI blog so let's look at it:
the only eval targeting research shows a ~5 point improvement between opus 4.8 -> mythos, but opus 4.7 -> opus 4.8 was a 4 point improvement. obviously not the same if the 5 point improvement led to solving significantly harder tasks, but then, let's be transparent about this evals and make it more details: difficulty filtering, example of what it could look like from public library?
the other AI R&D capabilities evals in the system card are actually not relevant anymore according to anthropic's own words:
"Claude Mythos 5, like Claude Mythos Preview and Claude Opus 4.7, exceeds top human performance thresholds on all but one of these tasks. The suite therefore no longer provides evidence that the model's capabilities are short of our risk thresholds"
only one that is not saturated is the "Novel Compiler" one, if you look at LLM training one (which they consider saturated) it's about how much the model can speedup the training of a small model on a CPU, i don't think anyone would say this is a good proxy for taking a decision to restrict capabilities of the model for ai researcher
idk honestly this feels wrong at so many levels
