ceo @luminal_ai delivering superintelligence to the proletariat.
San Francisco, CA
Joined June 2015
- Tweets2K
- Following519
- Followers3.5K
- Likes1.2K
Joe Fioti retweeted
Watch Luminal’s CEO, @joefioti give a talk at the YC Paper Club about how our specialized inference compiler allows our team to stay lean while serving Trillions of tokens per day
This week’s Paper Club is all about inference.
For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade.
But inference is much more than calling model.forward. Every token has to move through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality.
We brought together researchers and builders from @togethercompute, @ollama, @luminal_ai, and @sgl_project & @radixark to take us through that stack from end to end. We’ll see how a token gets from an API request to a GPU and back, why the same open model can perform differently depending on who serves it, and how new approaches to compilers and caching are making inference faster and more efficient.
00:00 - Why inference matters more than training
08:24 - Three kinds of inference innovation
12:14 - Dan Fu (@realDanFu): the lifecycle of an inference request
42:18 - Jeff Morgan (@jmorgan): not all open model tokens are the same
1:04:31 - Joe Fioti (@joefioti): compiling models with search, not heuristics
1:30:30 - Liangsheng Yin (@lsyincs): SGLang and memory for the agent era
Had a blast talking about search-based compilers at inference night at the YC paper club!
This week’s Paper Club is all about inference.
For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade.
But inference is much more than calling model.forward. Every token has to move through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality.
We brought together researchers and builders from @togethercompute, @ollama, @luminal_ai, and @sgl_project & @radixark to take us through that stack from end to end. We’ll see how a token gets from an API request to a GPU and back, why the same open model can perform differently depending on who serves it, and how new approaches to compilers and caching are making inference faster and more efficient.
00:00 - Why inference matters more than training
08:24 - Three kinds of inference innovation
12:14 - Dan Fu (@realDanFu): the lifecycle of an inference request
42:18 - Jeff Morgan (@jmorgan): not all open model tokens are the same
1:04:31 - Joe Fioti (@joefioti): compiling models with search, not heuristics
1:30:30 - Liangsheng Yin (@lsyincs): SGLang and memory for the agent era
How does a compiler think about running a model on multiple devices?
One common approach (and one we use in Luminal) is the Single Program Multiple Data (SPMD) paradigm.
The basic idea is to compile one local program, with communication operations, and distribute it amongst all devices. Then at runtime each device's local program hooks into different data, and all devices execute independently, only synchronizing at communication points.
The benefit of SPMD is a simplification of the execution model. We can think only of the local view of one device, with a consistent sharding strategy, and know our whole device mesh will run correctly.
This simplification greatly helps the compiler to focus on optimizing just a local program, without bringing in the possibilities of running different programs on different devices. The alternative to SPMD is MPMD (Multiple Program Multiple Data) where truly different code is ran on each device. MPMD is a strict generalization over SPMD, and allows for more complex computation patterns (like running a speculator on one device, or mixing devices of different capabilities in the mesh). However it greatly increases the compiler's complexity.
I am starting a robotics company in my spare time called Big Gripper™️ that takes cheap servos, webcams, metal, plastic, hooked directly up to Luminal Cloud, and lap all the custom robotic model companies out there.
Might point codex at stripe atlas and let it rip.
We are open-sourcing V1.0 of our agentic robotics framework, ASPIRE, with full real robot skill discovery and self improvement capabilities! On the day Astra arrived, our lab switched the backend model and saturated all the benchmarks to ~99% over night.
And the frontier is already on the move:
- Opus 5.5 slightly edges out Astra on our robot simulation bench.
- Sonnet 5.5 hits the Pareto frontier of cost vs. robot capability, $12/task.
We first introduced ASPIRE and /robot skill library in June, way before the big models could reliably do robotics. A rising frontier tide lifts all boats, but Astra was a 0->1 tsunami on many tasks.
Now you can also feel the Physical Bitter Lesson at home like we did on that fateful day ;)
links in thread
Is there a real moat for all of these sandbox companies? Feels like everyone is pivoting and the product is always the same???
- Fast boot time
- High clock speed
- Scale up and to zero
But like beyond lightweight microVMs it feels like the technical depth is fairly shallow? Like everyone is running on the same CPUs, and you can't really optimize the actual workloads because its just random agent-generated code at runtime.
Aside from race-to-the-bottom on pricing, how do you compete?
growing szn. need compute.
We're growing again! Doing over 50 trillion tokens a month @luminal_ai. If you can get me access to B300s in the next week or two, you'll be my best friend
We've been growing insanely fast on private dedicated deployments and are just now starting to open up our cloud to the public.
~2.3 trillion tokens per day.
Not happy till Luminal-compiled inference drives a few quadrillion tokens per day.
Might be a moment of hubris for me but I’ll still say it: skill issue. Noam and co had a really innovative model arch, and it’s wild to me they couldn’t figure out how to spend less on compute than they were charging users.
Modern batch inference is insanely efficient.
Just raise prices and use smaller models.
I pulled the Character bankruptcy docs. They show:
- They were spending $14 million/month on compute alone at a time their revenue was around ~$1 million/month
- Currently losing $9 million a month
- They admitted they enshittified & that it didnt work
404media.co/character-ais-ba…
I keep talking about this. High-batch inference is already are nearly as efficient as the human brain, and we’re on track for at least another 10-100x efficiency gain.
luminal is serving ~45 trillion tokens per month, and 5 million images per month.
we run more than an exaflop of compute.
still feels insane to me.
