Scaling PreTraining @reflection_ai, ex-Core Gemini ♊, Science of Scaling, Model Design, PreTraining 🤝 RL, Intelligence 🤝 Efficiency, Science 🤝 Infra.
San Francisco, CA
Joined May 2011
- Tweets389
- Following1.1K
- Followers402
- Likes1.8K
Pinned Tweet
What a year @GoogleAI (Dec 2022-Dec 2023)🚀Working with an amazing team all over the globe has been a highlight, impressed with how Gemini was built as a startup within Google. Been a unique rewarding experience with tons of learning along the journey. Another step forward in AI.
I’m very excited to share our work on Gemini today! Gemini is a family of multimodal models that demonstrate really strong capabilities across the image, audio, video, and text domains. Our most-capable model, Gemini Ultra, advances the state of the art in 30 of 32 benchmarks, including 10 of 12 popular text and reasoning benchmarks, 9 of 9 image understanding benchmarks, 6 of 6 video understanding benchmarks, and 5 of 5 speech recognition and speech translation benchmarks. Gemini Ultra is the first model to achieve human-expert performance on MMLU across 57 subjects with a score above 90%. It also achieves a new state-of-the-art score of 62.4% on the new MMMU multimodal reasoning benchmark, outperforming the previous best model by more than 5 percentage points.
Gemini was built by an awesome team of people from @GoogleDeepMind, @GoogleResearch, and elsewhere at @Google, and is one of the largest science and engineering efforts we’ve ever undertaken. As one of the two overall technical leads of the Gemini effort, along with my colleague @OriolVinyalsML, I am incredibly proud of the whole team, and we’re so excited to be sharing our work with you today!
There’s quite a lot of different material about Gemini available, starting with:
Main blog post: blog.google/technology/ai/go…
60-page technical report authored by th Gemini Team: deepmind.google/gemini/gemin…
In this thread, I’ll walk you through some of the highlights.
Samira Daruki retweeted
Reflection's first technical report is now available on Papers with Code!
Chat with it here: paperswithcode.co/pdf/123299
Beam’s model technical report is now available. Read the full report to learn about the methodology our team used to train Beam end-to-end in-house, here: reflection.ai/documents/beam…
Samira Daruki retweeted
Beam's 1.0 156-page tech report by @reflection_ai is out. Here's the training recipe at a glance 👇
– 23.8T tokens (13.7T unique × 1.73 epochs) on 6,144 GB300s for 21.9 days, 21% MFU
– AdamC, not AdamW/Muon: weight decay ∝ LR² → 0, peak LR from a fitted law η ∝ N^−α·D^−0.32·B^0.5
– Async RL with no staleness bound (avg 5 policy versions behind), 100M+ rollouts on ≤10.5K GB300s
– Agentic + alignment experts merged via full-vocab on-policy distillation
– Stability: depth-scaled residuals (1/√n), σ-gated attention, busiest MoE expert ends at 1.04× uniform
📄 Report: reflection.ai/documents/beam…
📝 Blog: reflection.ai/blog/introduci…
🤗 Weights: huggingface.co/reflection/Be…
Samira Daruki retweeted
Reflection’s 156-page Beam report is worth digging into: 23.8T pretraining tokens, then four weeks of RL on 10.5K GB300s generating 100M+ rollouts, followed by multi-teacher on-policy distillation.
The infra details are just as interesting: swapping the RMSNorm kernel reportedly improved speed by ~10%.
Beam’s model technical report is now available. Read the full report to learn about the methodology our team used to train Beam end-to-end in-house, here: reflection.ai/documents/beam…
Samira Daruki retweeted
thoughtful work in pre-training. they test every recipe change on a ladder of small models (about 300M to 6B active params) before trusting it for the final run but the ladder is only useful if the small models face the same conditions as the big one.
like this approach differs for data. if taken a small ladder model trained on (lets say 238B tokens which is 1% of the final run). if you sample its data from the full 13.7T pool, it never sees anything twice and the pool is about 58 times bigger than its budget. so the small model is always training on fresh data and the big one is not.
they fixed it with data-limited subsampling. for each ladder size, shrink each source’s pool by the same factor as the training budget. At 1% scale, the code pool becomes about 36.5B unique tokens, and the small model trains on about 84B code tokens (2.3 passes, same as the final run).
Beam’s model technical report is now available. Read the full report to learn about the methodology our team used to train Beam end-to-end in-house, here: reflection.ai/documents/beam…
Samira Daruki retweeted
Technical report is out. Covers a lot of detail on how Beam was trained.
Beam’s model technical report is now available. Read the full report to learn about the methodology our team used to train Beam end-to-end in-house, here: reflection.ai/documents/beam…
📑📊🗞️
Beam’s model technical report is now available. Read the full report to learn about the methodology our team used to train Beam end-to-end in-house, here: reflection.ai/documents/beam…
Samira Daruki retweeted
We've posted a 156-page tech report on Beam! We have good systems and it's worth reading. It covers all phases through pre-training, mid-training, SFT, RL, and OPD and goes into the infra, data, and scientific decisions we made throughout the process
📚 reflection.ai/documents/beam…
We've released the weights for Beam!!
🤗 huggingface.co/reflection
📄 Model card: reflection.ai/beam-model-car…
🧪 Try Beam in our playground & get an API key: platform.reflection.ai/playg…
📚 AND we take open model safety seriously and have posted our safety report: reflection.ai/beam-safety-re…
Samira Daruki retweeted
18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.
And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.
So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?
⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.
Read it here: huggingface.co/spaces/huggin…
Samira Daruki retweeted
I made a 3D visualization of Ripple-65B Preview's architecture! Check it out here (with an easter egg): shayantalaei.com/ripple-65b/
Ripple is a dense model with 80 layers and a width of 5,120, so its aspect ratio is 64. That makes it very deep for its width. In a model this deep, each layer keeps adding its output to the residual stream, and by the deeper layers the original token information gets diluted.
Most recent architectures handle this by changing the residual stream itself, like DeepSeek's manifold-constrained hyper-connections (mHC) or Kimi's Attention Residuals. We went with value residuals instead. Every layer mixes the original token embedding back into its attention values, and a learned weight in each layer decides how much.
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.
Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.
Learn more: ricursive.com/ripple
Samira Daruki retweeted
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.
Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.
Learn more: ricursive.com/ripple
Samira Daruki retweeted
RSI starts with infra 🔁 mlsys is in a golden age. agents are great researchers now. with tokens, "low-hanging fruit" is everywhere.
agents optimize speculative decoding for parallel drafters (DFlash2, DSpark) without training or quality loss. their 12-bit format beats cuBLAS.
Samira Daruki retweeted
Muon achieves lower loss than AdamW after pretraining, but does its advantage persist after fine-tuning?
Surprisingly, the answer depends on pretraining data scale: beyond a sufficiently large pretraining data budget, AdamW outperforms Muon after fine-tuning!
Samira Daruki retweeted
arxiv.org/abs/2611.03633
If pretraining TPP becomes larger then Adam could be better than Muon on finetuning - pretraining loss tradeoff. Really?
Samira Daruki retweeted
We're proud to have collaborated on a new benchmark built by @GoogleAI: Physical-Engineering's Last Exam (PLE).
Physical engineers design the skyscrapers, power grids, and chip fabs we rely on, yet no existing benchmark tested whether AI agents could do that work end to end.
Together with Google and practicing engineers, we built 85 long-horizon tasks across mechanical, electrical, civil, and structural engineering. On the private test set, all frontier models score below 50%:
1. Claude Opus 5.5: 48.3%
2. GPT 6 Astra: 45.8%
3. Claude Sonnet 5.5: 43.5%
4. Gemini 4 Argon: 40.6%
5. Claude Opus 5: 40.0%
On the tasks we co-authored, the top agent satisfied 91% of individual constraints, but only 41.3% of its deliverable items passed every hard safety check.
Learning how AI fails at physical engineering can help us improve future models and build a safer world.
This benchmark is also an open competition. Bring your own agent setup and submit it for evaluation on our private test set at ple-bench.com
Samira Daruki retweeted
Test of time award for sure
There's a new kind of coding I call "vibe coding", where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It's possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper so I barely even touch the keyboard. I ask for the dumbest things like "decrease the padding on the sidebar by half" because I'm too lazy to find it. I "Accept All" always, I don't read the diffs anymore. When I get error messages I just copy paste them in with no comment, usually that fixes it. The code grows beyond my usual comprehension, I'd have to really read through it for a while. Sometimes the LLMs can't fix a bug so I just work around it or ask for random changes until it goes away. It's not too bad for throwaway weekend projects, but still quite amusing. I'm building a project or webapp, but it's not really coding - I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.
Samira Daruki retweeted
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.
GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
Samira Daruki retweeted
Finally releasing the results of the arch ablations I was running with bux over the weekend
Samira Daruki retweeted
Multi-Token Prediction (MTP) can be done much better.
🚀 Turn out,you can recover the gradient for KL self-distillation to next-token prediction (NTP) for free without ever computing the KL. You just need a one-line code change:
h_ntp += h_mtp.detach() - h_mtp
Soft Multi-Token Prediction is pretty much a free lunch for MTP.
📜 Paper: arxiv.org/abs/2610.38331
💻 Code: github.com/JaydenTeoh/Soft-M…
🤗 HuggingFace: huggingface.co/papers/2610.3…
ALT Soft Multi-Token Prediction Illustration
Samira Daruki retweeted
Really Cool work.
Pre-training, supervised fine-tuning, and reinforcement learning can be unified under a single likelihood-based optimization framework.
I have not been able to make it work on scale yet : )
New work on Training Foundation Models with a Single Algorithm:
Paper: arxiv.org/abs/2610.40564
Code, Models: yudasong.github.io/unified-t…
This work introduces Unified Training, a framework that unifies pretraining, midtraining, supervised fine-tuning and RL under a single maximum-likelihood objective. Building on MaxRL, it enables static data and RL-generated data to be trained together within the same system.
Empirically, Unified Training improves accuracy and compute efficiency, generalizes better to harder reasoning problems, and reduces forgetting.
Interestingly, reusing data across training stages can improve performance even when conventional staged training fails. It also produces models that adapt more effectively to new tasks, leading to stronger continual learning.
See an excellent thread by @yus167.