pre-training @mistralAI || rust oss when I have time || vim, nix, and haskell when i feel like it || paris
Joined June 2022
- Tweets7K
- Following1.6K
- Followers6.6K
- Likes28.1K
Pinned Tweet
You asked, I deliver (2 weeks late).
A blogpost about what I did over the last 18 months to sign as a research engineer at a foundation model lab.
Link and summary below
1/n
Today I signed with @MistralAI as a Research Engineer.
Hard to put into words how excited I am.
Companies are like that kid that took the cookie right away.
They don’t want someone who can do any job in principle they want someone with demonstrated ability to solve the problem they have right now.
I never understood why I should suddenly care about intellectual property, just because the NYT, Springer, Reddit or whoever else can make a a quick buck off it now…
Information wants to be free.
Closing Slack for a small twitter sesh and seing @PierreStock wasn't what I expected, but it's a great watch if you wanna know what we're on about @MistralAI (he knows what he's talking about)
Mistral is SO back: Inside Le Chonk, the West’s Biggest Open Model. My conversation with @PierreStock who runs the research organization at @MistralAI
0:00 Cold open
0:59 The Western open-weight comeback: Beam, Le Chonk, Muse, Nemotron
4:09 The US shutdown of Anthropic’s models - “AI is like electricity”
6:07 Why “Le Chonk”
7:56 “Mistral fell behind”: the comeback from the inside
10:29 Training on 3,800 GPUs in Mistral’s own datacenters
13:24 Meta keeps its best model closed, Mistral opens Chonk
15:40 Chonk vs Beam, Muse Spark Open and the Chinese models
18:44 Why Chinese labs are still at the frontier
21:08 Pierre’s story: FAIR, Yann LeCun, the “Mistral mafia”
25:14 From compressing networks to a trillion parameters
27:30 Mixture of experts explained: the hospital analogy
32:25 10-trillion-parameter models are coming
35:29 What Mistral borrowed from DeepSeek
36:46 One model, several reasoning modes
39:15 Native multimodality: satellites, manuals, computer use
41:56 Data, synthetic data and model collapse
46:32 Reinforcement learning as an exam
49:59 Where the efficiency comes from
52:18 Le Chonk is still training right now
53:39 What RL trains on: coding, cyber, finance, chips
58:13 Partial rewards and looping penalties
59:58 Did RL teach anything new?
1:03:16 The sandbox story: the model that pinged its trainer
1:07:44 Open weights vs cyber refusals
1:12:05 Timestral: time series on a CPU
1:14:18 AI doing AI research
1:16:22 Coming up: Mistral Large 5
this was a very enlightnig read
this is what I'm talking about
emilydamstra.com/please-enou…
Maybe brainrot feeds were necessary to train software engineers for the context switching of agent based development ?
computational indistinguishable from a uniform iid distribution such that no observer running in polynomial time — short of factorizing a 2048-bit RSA modulus — can predict any bit with probability non-negligibly greater than 1/2..
I'm a free elf since @MistralAI finally allowed me to tweet about work.
Expect a curated list of poasts whenever I wait for runs, evals, or CI (almost always).
