Eric Zhang retweeted
fine-tuned qwen on my 16gb gpu to respond without the letter e
what i learned:
> simply masking the 'e' tokens out produces gibberish
> you can just RL a model to speak coherently
writeup here: kevin-wu.ca/blog/qwn/
supergoats right here
Spent some time creating a lightweight, performant async RL stack in pure JAX with @AdityaMakkar0! We share a work log covering inference, RDMA weight transfer, memory optimizations, sharding and a few more JAX engineering insights we've seen along the way! It’s a bit of a long read, but I hope it sheds some light on the systems side of scaling RL!
Eric Zhang retweeted
It's my last week @modal.
This summer, I had the chance to work alongside @waltergoat, @colin_weld, @paulgb on our Sandbox product. It was my first time working on systems at this scale, and I couldn't be more grateful for their mentorship.
Sandboxes are full of gnarly distributed systems problems. I wrote about one of them - distributed locking - on my blog: scotthao.com/writing/distrib…
To ship it, I needed to break rules for practicality and performance. This led me to formally verify the design with TLA+ to catch the many race conditions.
This is also my first blog post! I don't share my writing often, but I'd like to do it more. Hope you enjoy.
Eric Zhang retweeted
We’ve open-sourced the MoE megakernel we use to train models on NVL72s. A few of my favorite details: bf16 and mxfp8 support, pull-based dispatch, configurable overlap granularity, full determinism, and no CPU-GPU syncs. Read more in our blog and code!
We're open-sourcing Mixture-of-Kittens (MoK), our MoE training megakernel for NVL72s.
It fuses all Mixture-of-Experts communication and computation into a single, fully deterministic kernel, and runs up to 2.37x faster than the strongest public baselines.
ALT Bar chart: Mixture-of-Kittens reaches up to 2.37× higher MXFP8 forward throughput than the fastest public baseline on GB300 NVL72s, across Kimi K2.7, GLM 5.2, Qwen 3.5-397B-A17B, and DeepSeek V4 Pro.
Eric Zhang retweeted
A billion people produce the most valuable dataset in the world every day – and delete it every night.
We’re recording it.
Introducing @AttentionInc
Eric Zhang retweeted
Tech has decided that regulation is the enemy of progress, but innovation requires capital
Regulation builds the trust necessary to inject billions into markets
To enable free innovation, Andera raised a $37M series A led by Lightspeed to scale financial oversight with agents
Eric Zhang retweeted
This is my first @FIFAWorldCup where I can’t watch the games because adulting and employment👴
So instead I find myself alt-tabbing in every 5 minutes or whenever someone texts me "YOOOOO" to check the score 😭
So I built Crossbar! The world cup scoreboard living in your Mac notch, with live game updates and replays natively so you never miss a beat! ⚽️
100% free & link in comments!!
Eric Zhang retweeted
Excited to share some of the work we’ve been doing at @gen_intuition! Together with @kyutai_labs and in collaboration with Epic Games we introduce MIRA, a real-time multiplayer world model! If you’re at ICML 🇰🇷 feel free to drop by Booth 111 to chat more about MIRA and world-models!
Introducing MIRA.
A playable, multiplayer world model.
A dream of Rocket League.
Trained on 10k hours of data collected with publicly available bots, MIRA learns the dynamics of a four-player game. The model runs in real time at 20 fps, based on the keys you and the other players press.
Built by General Intuition and @kyutai_labs, in collaboration with Epic Games. Not used to develop Rocket League.
▶️ Play the demo, read the technical report, and explore the open-source code at generalintuition.com/mira
At ICML? Find us at Booth 111 to try it yourself and dig into the results with the team.
Eric Zhang retweeted
A super long overdue (3+ years?) post on scaling laws.
Compute is expensive. Scaling laws are a way to help us reason about the optimal compute allocation between data and model size before committing to a large run.
The post covers what scaling laws predict, how compute-optimal allocation works, why Kaplan et al. and Chinchilla disagree, and how data limits + fitting details make extrapolation tricky.
lilianweng.github.io/posts/2…
Eric Zhang retweeted
the riskiest live demo i’ve ever pulled off:
2000 phones synchronized with light and audio down to the millisecond.
i'll never forget this night.
👀
Join us in welcoming the first cohort of Delta Fellows! 🎉
Congrats to the ~100 amazing researchers and engineers joining the Delta Institute family. We're excited for our fellows to get to know each other through dinners, retreats, and much more!
Our fellows come from diverse backgrounds: undergrads, PhD students, high-frequency trading, big tech, startups, neolabs, frontier labs, and more. What brings them together is their kindness, intellectual curiosity, and intrinsic passion for their field.
deltainstitutes.org/cohort1
this man is my inspiration
i don’t usually share when things go wrong
but recently i was researching world models and it went sideways, but i learned a lot
i wrote a new essay about how not to do research rajan.sh/multiplayer
2
13
2,886
first years at wloo just launched this when i walked into the room
Think you’re smart? Prove it.
We built the ranked Human Benchmark for uni students.
Try it out --> goosetrials.com
P.S. Ur absolutely cracked if you can beat my Tower of Hanoi
