CS PhD student @HKUniversity, supervised by @difanzou.
Joined November 2024
- Tweets11
- Following54
- Followers10
- Likes7
Interesting finding from Mistral Large 4: PPL correlates well with knowledge-based evals, but much less so with agentic performance (SWE-bench Verified).
If PPL isn't a good predictor of agentic potential, what should we be looking at instead? 🤔
Today, we release the weights of a new and improved Le Chonk, advancing the open frontier for agentic coding and cybersecurity.
💻 Built to build: Le Chonk ranks #1 on @cognition’s FrontierCode among open models, and sets a new benchmark for Pareto efficiency on @artificialanalysis.
🛡 Built to defend: State-of-the-art in cyberdefense, put through the paces by the world's leading cyber vendors, including @CrowdStrike, @cloudflare, @rubrik, @zscaler, and others.
Built for sovereignty: Make it yours. Download the weights on @huggingface , customize it with Mistral Forge, and deploy it wherever you need it.
Read more from our team and customers 👇:
27
doodle retweeted
Muon now trains frontier LLMs. But how does its spectral orthogonalization shape feature learning?
New paper: on factual recall, Muon learns facts in a more balanced way and is far less slowed by softmax saturation.
🔗arxiv.org/abs/2610.02798
[1/8]
(1/6)
Weight decay has a direction, too.
We usually tune its strength. For matrix weights, we can also choose how shrinkage is distributed across singular modes.
New blog with my advisor @difanzou :
Rethinking Weight Decay
doodlext.github.io/blog/reth…
(5/6)
Nearly the same pretraining loss, different internal geometry.
Ordinary WD vs asymmetric decay (p=0 Attention, p=2 MLP):
Loss gap: 1.47e-5 nats
MLP weight stable rank: +40.3%
Similar prediction loss leaves room for different spectral structures.
51
(4/6)
Case study: head-wise Muon.
Updating V as a full matrix improves validation loss over splitting V by head. Adding p=2 decay on V further improves the default full-V recipe:
3.01968 -> 3.01596 -> 3.01324
Update geometry and decay geometry offer two tuning choices.
50
(3/6)
Does this spectral preference survive training?
We start with an exactly solvable linear model, then examine an 8-layer Transformer. The spectrum-shaping effect remains visible after task-driven training, with different responses across matrix types.
77
