@southparkcommons (early OpenAI, Google Brain, Meta) https://xitter.cf/t.co/bxo5udY3ib https://xitter.cf/t.co/SLix8Hrt4w https://xitter.cf/t.co/Ur3GWKpmp6
San Francisco, CA
Joined February 2011
- Tweets2.5K
- Following1.2K
- Followers11.7K
- Likes1.5K
Pinned Tweet
Gradients are energy-inefficient due to long-range dependencies, yet we lack a viable alternative. What if we crowdsource the discovery of what's next?
Starting a weekly in-person reading group in SF. WebGPU, online learning, Hinton. Email [email protected].
Talked about the reasons to get rid of backprop last month. TLDR; memory wall.
Yaroslav Bulatov (@yaroslavvb) wants to get rid of backpropagation.
For Episode 7 of The Interlude Show, @jakekang and I chat with Yaroslav, who worked at Google Brain, early OpenAI, Meta, and Together AI. He now runs the Sutro Group at South Park Commons (@spc), exploring energy-efficient learning algorithms with AI agents.
We get into why moving data matters so much for energy use, how the memory wall changes algorithm design, and how gradient checkpointing can make a workload faster even while doing more calculations. We also discuss early OpenAI, his skepticism about reinforcement learning, and why recursive self-improvement still faces new bottlenecks.
We close with whether more code translates into better outcomes, and the difference between AI that helps us pursue our goals and AI that gets better at capturing our attention.
Chapters:
00:00 - A plan to rethink backpropagation
00:49 - Introducing Yaroslav Bulatov
01:30 - Energy-efficient learning and the 20-year research plan
03:25 - Why focus on software now?
08:05 - Sutro Group: using agents to explore learning algorithms
10:57 - The memory wall: arithmetic versus data movement
17:07 - Why peak memory is only part of the energy story
18:40 - Why Yaroslav wants to replace backpropagation
24:27 - Mixture of experts and disentangling neural networks
27:24 - Toy problems: matrix multiplication, sparse parity, and MNIST
31:21 - Early OpenAI and skepticism about reinforcement learning
34:41 - Gradient checkpointing: doing more math to move less data
38:51 - Power grids, intelligence per joule, and new bottlenecks
43:25 - Do more tokens and pull requests mean better outcomes?
46:21 - Superintelligence, specialization, and physical limits
51:49 - Creative freedom, attention capture, and possible AI futures
Reading Evan's post this morning I'm reminded again of "buy vs rent" math. Neoclouds margin is derived from the price of risk, so unless you have a better source of financing, renting could be cheaper -- substack.com/home/post/p-217…
Found a fun use for Thermal Master P3 camera, track heat propagation when you schedule computation on different parts of the chip spacesheep.dev/@yaroslavvb/e…
Wondering if there's a kind of Moravec's Law for agents. Difficult things are now trivial. And yet last week I spent billions of tokens + personal hours getting Claude to symlink some config files from the right places
My roommate just launched this. Share html files without Anthropic/OpenAI lock-in. I've been using for pretty 100% my output. Like this interactive visualization of the chip I've been playing with -- spacesheep.dev/@yaroslavvb/e…
My agents sent me 355 hours of reading this week. I can't read that fast.
We're launching Spacesheep (YC F26) to fix that: agents' work lands in a space your team can actually understand and work on together.
Try it: tell any agent "share this to spacesheep.dev"
Bearish case for local models:
- intelligence/$ better in a data center.
- network: you can stream Battlefield over GeForce, fully rendered remote frames at 60 fps
- security: quants and TSMC already keep secret things in the cloud
On the other hand, cell networks in SF are worse than in Rwanda (I checked during ICLR 2024), maybe still need on-device intelligence
run in with a data integrity enforcement official at the @GPU_MODE meetup yesterday
Been following this group's work for a while, had a suspicion that advanced optimization methods would show greater benefit on small models as it's a harder optimization problem, happy to see this release huggingface.co/collections/n…
Tried Gemini 4 Argon and it worked really well. I plugged in Thermal Master P3 and instructed to make a custom live-streaming app with OTA update functionality over an unsecured network. It one-shotted it. Livestream -- spacesheep.dev/@yaroslavvb/t…
A high-level view of the things it had to do to reverse engineer it -- spacesheep.dev/@yaroslavvb/r…
Summarizing some digging on the origins of the memory wall spacesheep.dev/@yaroslavvb/i…
Spent last two weeks learning electronics by having Opus run experiments on the Esperanto AI chips and condensing the results into interactive diagrams like this spacesheep.dev/@yaroslavvb/e…
Compiling some "numbers you should know" for Human Brain vs H100 .... 80B transistors vs 86B neurons, number of transistors flips ≈ number of synaptic arrivals spacesheep.dev/@yaroslavvb/b…
Enjoyed watching Naveen's talk at our Mondays "energy efficient" AI meetup. Tldr; Existing AI stack is built upon 80 years of abstractions, due for an overhaul.
My talk from @theallinpod conference. You can learn a bit more about what we're up to @unconvAI !
Have been experimenting with a mobile way of reading papers -- Listen to 10 minute summary, then give agents GPUs and ask clarifying questions while walking around. They run experiments and produce phone readable artifacts which can be checked on the go github.com/yaroslavvb/gradie… Enjoyed @m_elhoushi Don't Drop Dropout, seems like a good paper alphaxiv.org/abs/2609.05275
TLDR; layer dropout appears to work if you do the following:
1. start at high dropout and gradually lower it
2. drop-out top layers more often than bottom layers
3. tune hyper-parameters properly.
Properties 1 and 2 turn this into an approximation of stochastic growing, and an added benefit is that a subset of layers (probably the bottom subset) works as an independent speculator model for speculative decoding.