Researcher @EpochAIResearch | Prev: Inference @ NVIDIA , CentML Just here to see the latest rumours in AI personal opinions only
Joined September 2023
- Tweets162
- Following561
- Followers143
- Likes5.2K
Pinned Tweet
Prediction: AI is going to have a 2000 style bubble bust in the next 2 years but we still get AGI* by 2033.
*AGI defined as TED-AI from AI-futures project
Jason Li retweeted
There is an absolute barrage of inference providers popping up that get to $1M/mo extremely quickly.
This is the rough economics of inference providers:
Even if you rent on-demand B300s at $8/GPU-hr, many open source models (GLM 5.3, Kimi K3, DeepSeek 4.1 Flash / Pro) can get you >$10-20/GPU-hr at reasonable throughputs (100tok/s). DeepInfer from DeepSeek directly shows they can get $38/GPU-hr on off-peak hours. Every chip / model has a different throughput / interactivity curve, but there is a lot of arbitrage between even high rental spot prices and token prices. New chips (Vera Rubins) also change this meaningfully. Your ability to be making gross margin relies on your utilization and how good your stack is.
Gateways like OpenRouter, Vercel and more help solve that problem by aggregating demand to you. May inference providers discount their tokens to get longer term commits too. I've even heard stories of selling back Chinese model tokens to Chinese labs!
One thorn in the economics of these players is that many Chinese open source model labs are now asking for rev share if you're at a certain scale, espp for pre-release access to their models for day 1 support. That further pushes the threshold to be in the money.
The barrier to entry seems low and falling. There are even stories of smaller ones who can "vibe code" a way to stand up OSS models on a cluster they rent. And as a result, we're seeing 10s of such companies pop up almost every week and immediately skyrocketing revenue at low-ish gross margin and raising rounds on the back of that.
Unsurprisingly, because the actual service provided seems commodity, this eventually drives up compute prices and drives down token prices. In theory, it would seem like neoclouds who have the lowest cost of compute and themselves have started inference businesses would win the race to the bottom (Crusoe Coreweave Nebius and GMI all have inference businesses), with larger players (Modal Fireworks Baseten) being second because they are well capitalized to make bigger compute commits cheaply and have demand (high utilization), but the reality is the pie is growing so quickly that even upstarts are absolutely crushing it and growing aggressively (inference dot net, DeepInfra, Wafer, Relace, Morph).
Strategies vary across the board: neoclouds want to serve inference, inference players want to be neoclouds and do higher margin reserved instances / post training, and some want to go heterogeneous hardware or creatively flexible compute constructs.
The risks are: do open source models continue to be competitive? Will their creators rent seek more? Will their shape change compute needs? How do different chips change economics? Who gets compute in a supply crunch?
Below is a aggressive sketch of how gross margin scales with utilization for various models / chip sets.
For now, there are many inference providers and they’re all ripping!
Jason Li retweeted
As someone with a portfolio of energized land assets here is what I am seeing:
1. Neoclouds are disorganized and poorly capitalized. Some are definitely better than others however.
2. Hyperscalers are intractable, bureaucratic organizations that take 3 months to do an NDA and want 18mo escrow periods. We built our portfolio by competing against these terms because of our speed and simplicity. The more future assets orgs like ours win, the bigger the problem for the hyperscalers downstream (see 3).
3. There are delays in the existing pipeline that will catch people off guard in 2027. Hyperacalers will (obviously) react by focusing on their own internal compute needs over any third party. If the third party is a model company, they will tell them to go through their model router where they will also take a non-trivial vig. Them telling Ant they have no compute for them (rumored in Ant’s S-1) if true is a huge tell that this is already happening.
All of this is to say that it does look like there will be a lot of partially stranded compute and compute sitting in inventory in 2027 than is currently forecasted. This is bought and paid for but effectively underearning (below total cost of capital) or collecting dust. We see this as true for everyone - model companies, hyperscalers, neoclouds except ONE (you can guess which one).
This dislocation will benefit two kinds of orgs: the non-bureaucratic big scalers who can build industrial scale quickly and the nimble, well capitalized players feeding them.
2027 will be a turbulent year because it will clear out a bunch of wrong/bad bets. It will also be a consolidation year for a few.
Jason Li retweeted
CITADEL’S KEN GRIFFIN: WE’RE ALMOST CERTAINLY IN AN AI BUBBLE
Billionaire Ken Griffin warns investors are “almost certainly” in an AI bubble, as companies struggle to turn massive AI spending into measurable productivity gains.
With Anthropic reportedly eyeing $750 billion in future computing expenditures, questions over AI investment returns are intensifying.
But Griffin remains bullish on AI’s long-term potential.
Jason Li retweeted
I can write pages and pages about the economic damage of the malinvesment, the froth caused by AI capex, the crowding out, and how rates were way too low
In my wildest dreams, none of it could’ve ever been as effective as this tweet
Jason Li retweeted
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
🔗 github.com/deepseek-ai/DeepI…
1/4
Jason Li retweeted
This is directionally consistent with our internal data @cognition. Many tasks within software engineering are now "intelligence saturated." Once any model can do the task well, cost and speed are all that matter.
OpenAI, Anthropic and other leading AI labs face a growing challenge: Better models aren't necessarily attracting more usage. A 10-point intelligence advantage now earns just 20% more developer traffic, down from nearly 69% in late 2025.
Full story: thein.fo/4hGJ9jY
SA has forecasted over 50GW of datacenter capacity added in 2027. Why is this bearish for AI infra?
To put that into a revenue perspective, their InfX revenue calculator shows that leading current-gen chips can achieve ~$50B/GW in annualized revenue (even with a 60% utilization assumption).
So 50GW could support $2.5T of annual API revenue before even accounting for improvements in next-gen chips like Rubin.
This means that even if only half is dedicated to paid inference, there's enough datacenter capacity to support over $1T in annualized revenue once equipped with GPUs.
So if instead of $1T in combined API revenues we're closer to $500B, there could be a severe supply glut at today's prices. Either compute goes underutilized or prices fall rapidly, cutting into margins.
Datacenters 2027: The Year of Acceleration,
+50GW in a Single Year
60GW Under Construction,
SemiAnalysis Leading Indicators,
1.3T of Lease Contracts Signed YTD,
The Shortage Could Move to Chips
newsletter.semianalysis.com/…
This is big, and it shows that TPUs can compete even on models not co-designed for them. Now I wonder how TPU v8i will perform as an inference-specialized chip.
BREAKING NEWS🚨: @Inferact & Google submit their FIRST preliminary TPUv7 Kimi K3 2.8T Model Submission, and it is already delivering strong perf per TCO against GB200 NVL72, B300, and MI355 in the ultra-high-interactivity regime due to their megakernel approach for external customers.
One of the most important aspects of an external compute platform is a strong community, and what makes this submission special is that Inferact (Makers of vLLM) led the submission end to end using the TorchTPU vLLM stack.👇️(1/6)🧵
Jason Li retweeted
Heard Anthropic recently has paused lots of training jobs(roughly 50% of their computation) and relaxed the quota on Fable to all in inference to pump the ARR.
They desperate for the IPO.
Let's wait for today's event to see if they want to offer a discount on the IPO valuation.
If they say "sir, would you like 25% off 2T? Or 10% more, please buy our shits"
What would happen to OAI's fund raising at 1.3T?
It'd be the disaster to AI names as the valuation discount shows they have no confidence.
Great explanation of what these recent inference chips (MAIA, Jalapeno, TPU v8i) actually optimize for.
So how do you make an efficient inference chip? Like most systems problems, it comes down to getting the most useful work out of a fixed power budget
Maia 300 is a cool example of getting significant wins in tokens/W by fitting to the compute constraints of inference. Unlike training and high-throughput inference, interactive inference is mostly memory bound. This enables chip designers to cut systolic flops (saving both power and area) while designing the architecture around inference dataflow to limit data movement
Looking at the specs, Maia is eerily similar to OpenAI’s Jalapeño. While both orgs clearly did some very cool things in the architecture, microarchitecture, and networking, the main win is HBM4 (which gives >2x bandwidth over HBM3) and better provisioning of resources
If you look at the FLOPs/W and bandwidth/W figure I scrapped together, Rubins will still be the best for compute-bound workloads (and combined with Nvidia’s superior software stack and reliability, they will likely maintain their monopoly on training)
However, if we estimate the ceiling of Maia 300 and Jalapeño at ~40% more power efficient than a Rubin on inference (in large part because they have similar HBM4 bandwidth but spend way less power on FLOPs) and they can bring up a reasonable software stack, there is a strong case to use Maia/Jalapeno-class chips for more power efficient memory-bound inference
Another side note for anyone interested in Groq and Cerebras chips who also claim inference-specialization, they actually cut out HBM altogether and provision much more area for SRAM, which hyper-optimizes for ultra low batch, low latency inference. They are moving in the direction of heterogenous disaggregation by using GPUs for prefill and even decode attention, while keeping their SRAM chips for exclusively decode. However, because of memory capacity limitations, these approaches struggle to keep up with growing model sizes and longer context lengths
I think we’ll see more chips along the Maia, Jalapeño, TPU v8i route that still uses HBM to keep weights and KV cache in a more power efficient way than SRAM, but provision chips correctly to not waste power on training optimizations
Takeaway - inference has become sufficiently different from training (and has carved out a majority of demand) that we are fundamentally rethinking our software stacks, chip architectures, microarchitectures, and resource provisioning. Its exciting to see the wins starting to pay off with meaning power efficiency gains!
Jason Li retweeted
I thought Gavin was coming back to his senses
then he hit me with 10 yr useful lives on GPUs
it's over
Jason Li retweeted
on track
AI predictions for next 6-9 months
-> Anthropic/ OAI fail to ban OSS in the US
-> Anthropic/ OAI continue to innovate product but stall on fundamental tech breakthroughs
-> SOTA Gemini 4 launches at 10x lower cost
-> Elon open sources Grok
-> Anthropic/ OAI ARR growth stalls as enterprises move from tokenmaxxing to rational token consumption/ OSS
-> slow realization in public markets that AI may not replace human labor in next 1-3 years/ has not had a fundamental tech breakthrough in over a year/ lacks ROI/ AI hyper and lab spend not durable
-> moonboy VCs/ hedge funds/ softbank continue to fund Anthropic/ OAI equity raises in private markets as public markets for them are now fully closed
-> CXMT, YMTC announce capacity expansions significantly above market expectations
-> hypers deliver good q326 results on first look but fail to show meaningful AI ROI, guide to "significant capex expansion in fy27"
-> broad realization in public markets the AI dream we got sold on is not anywhere close to realization
-> 10yr hits >5%
-> hyper bonds reprice >10%, CDS hit 3x current levels, and AI credit market dries up as market realizes GPU ABF was not a good idea
-> hyper equities reprice 30-50% lower
-> ORCL falls below $30 as rumors of bailout make the round
-> market realizes memory is cyclical and LTAs won't hold
-> 50% of korean traders get wiped within a week
-> hypers cut fy27 capex
-> semis reprice >80% lower
-> fed cuts >150bps to avoid recession
Good to see them make some concrete predictions. I thinks the Anthropic IPO will tell a lot about how on track we actually are to a trillion in revenue.
SemiAnalysis's Dylan Patel predicts OpenAI and Anthropic could reach $600–700 billion in combined revenue by December 2027--and raises the possibility of $1.5 trillion by year-end
“As far as AI’s impact on the U.S. economy, if AI revenue is only $3 trillion in 2031, then, yeah, we’ll have big problems.
“But I’m of the belief that we can get to $600 billion or $700 billion across OpenAI and Anthropic alone by December of next year.
“I’m of the opinion that you can get to $1.5 trillion, maybe even by the end of 2027, with just those two companies. So that’s the end of 2027. Then the payback is clear, right? You know, where they’re getting that from.
“Ultimately, the capacity that people are building out is matched to what they think business is coming to. There’s always going to be a mismatch, if we’re in the building phase, between how much infrastructure you’re spending on and the AI revenue, because today’s AI revenue is really based on the infrastructure you spent on in prior years.”