Pinned Tweet
Frontier performance at a fraction of the compute.
Today at @i14labs we’re launching Sugar Glider, our new family of LLM architectures!
Join the waitlist for early access: i14.ai
Sugar Glider delivers a new Pareto frontier: extreme memory and compute efficiency whilst maintaining frontier performance, even for large models (tested up to ~1T params).
O(1) per token inference → ✅
Fixed size KV cache → ✅
Frontier long-context retrieval quality → ✅
1 billion token long context retrieval → ☑️**
(**100% -> single needle @ 1B-token context, 90.5% multikey 1 @ 1B-token context.)
How did we do it? Turns out a few decades in the trenches of human memory and language model research is a pretty handy starting point to solve the “intelligence vs efficiency vs memory” conundrum.
... and just for something extra, we’ve been converting some of the most powerful and popular open-weight models to O(1) inference and fixed memory, with each conversion taking just hours :)
Tune in via our waitlist for early model access i14.ai
More results dropping soon!
Great write up. We were also similarly impressed with the Trainium training performance and upgraded tool stack in our research work with @awscloud
Our results in comments…
Amazon’s Trainium Takes a Strike at the Training King
Trainium Best for Inference?
But now good for Training too,
Native Trainium PyTorch, High Quality CI,
TorchTitan, GPU Impossible Profiler
newsletter.semianalysis.com/…
2
6
297
Hard agree. Think about when agents start building and spawning agents 🤔. Ability to comprehend and plan for these kind of exponentials isn't well understood when planning physical infrastructure.
We cannot physically serve a trillion agents with existing infrastructure.
The world doesn't have enough CPUs, GPUs, power, or space.
So we have two options:
1/ compress the workloads in software so agents do more with less
2/ or get more out of the hardware itself.
Probably both. And faster than anyone’s current roadmap assumes.
2
120
Fascinating insights into where new AI research is getting done 2025 vs 2022.
- Flip back from Industry to Academia
- Zilch coming from the frontier labs
- Deepmind out of the top 30
- NVIDIA comes in at #29
- USA from 8 out of top 10 down to 4
Source:carnegieendowment.org/featur…
2
106
FACT: Next generation of post VR chips actually create a black hole that swallows all light - fortunately you required the energy from that supernova’d star to power them anyway
2
90
Great look behind the token curtain from Sequoia’s @gradypb:
Observations:
1/ This is much more “normal” than I was expecting. I was secretly hoping these LP/VC interactions would have more of a Palpatine / Vader kinda feel to them. Next time.
2/ Architecture specialist Neolabs is where it’s at - especially the shoutout re: efficiency. Can hand on heart say from our own research work there is so much juice to be squeezed here. So much so that I do wonder about the duration of compute shortages
3/ “Own your own intelligence”. This is true - but unless you’re building tooling that allow companies to bring more of the frontier lab capabilities in-house, there will always be a large asterisk over “own”. Still need a bunch of 3rd party vendors to properly stitch this together.
4/ Nexus of #2 and #3 is where it gets super interesting. Big unlocks in efficiency and more and more companies don’t have to scour the world (and sell their firstborns) to get their hands on frontier-scale compute. The tooling to train, run inference and improve pareto optimal models becomes trivial and moves in-house. Experiments become “cheap”, parallelizable and fast to iterate on - market-winning models can be built by the relentless and resourceful <PM,Analyst,Engineer> rather than a massive lift from specialist teams.
The @BostonCollege Investment Committee (an LP and my beloved alma mater) asked for a few thoughts on what's happening in AI. I recorded a test run yesterday morning and then shared it with my partners, who encouraged me to share it more broadly... so here you go!
This is not a sales pitch, it's just a reflection on what we're seeing. And it wasn't intended to be shared, so please pardon the rough edges.
loom.com/share/c016702964a04…
2
114
Tom Mcleod retweeted
If you thought this was a bearish view on neolabs, let me write the bull case.
1. Compute is a great case of Helmer's cornered resource. There is a window of time where you can acquire a lot of compute and capital. In the future, both funding might dry out or compute prices might continue to rise. It could get very difficult to compete.
2. Big labs will suffer from innovator's dilemma. They're unlikely to:
a) deviate too much from their cash cow, coding
b) ship anything that cannibalizes revenue
c) go after opportunities that incrementally look too small (<$1B)
3. Empirically, many neolabs are working and generating significant revenue. They're just not being loud about it. Admittedly, it's still very much early days.
4. A lot of talent sees more financial upside in neolabs than big labs. Whether the companies show signs of life or fail entirely, there are at least 5-6 large promising acquirers.
All startups are difficult and the point of the original post was to illustrate specifically the difficulties of running businesses with massive upfront compute purchases, not say it's impossible.
The economics of a Neolab.
A neolab is loosely defined as a startup of AI researchers who raises a lot of money pre-production to be able to finance GPU compute to take on a large AI problem.
To buy 1000 GB300s or ~14 NVL72 racks will set you back $125-150M for 3yrs with 15-30% upfront. That’s about ~2-2.5MW.
Thats about enough to do 10^25 flops a quarter and get to a GPT-4 level model which is 1-2 OOMs off frontier for pretraining.
If you post-train on a great open source model, you have a better chance of getting to frontier. The risks are a) you need to spend millions on RL environments too and b) being lapped by another model release while being tied to a base model.
For this to payback, you need to give your customers a better and ideally cheaper inference service than a base model and serve them for long enough to recoup your large investment. Even at 50% margin on inference, to recoup $10M in training means serving ~10T tokens (!) if you price like Fable / Astra given a standard cache read / input / output split ($2/M blended). And you have to justify being better than a release like Opus 5.5 which is even cheaper. Often, you end up charging your customers a huge premium in terms of platform fees and compute fees on top of pure inference.
Meanwhile, every hour you’re not utilizing your GPUs you are burning money so you typically resell this compute back to a broker or run inference for open models / resell spot instances. At below a ~60% utilization on spot, you will still lose money.
Add to that insane cost of talent.
So what can you do with the compute?
- Not play the model game at all.
- Play an entirely different model game (Jev, World Labs) that if big labs played, would either a) cannibalize their business or b) be incrementally not significant revenue c) would cause too much distraction from the main main thing
- Acquire a proprietary data set (Peridodic Labs) in enough volume in a domain of usefulness to eclipse frontier quality. Often happens in robotics, biology, chemistry.
If you do overcome the challenge of building a model that is useful and well priced beyond big labs models, given the huge price of compute, you still need to play in an area where the revenue / compute ratio is signficant and market demand is large enough to payback your compute spend.
It is a difficult game.
Tom Mcleod retweeted
i14 Journal Club: Foundation Models Where Math Meets Cognitive Science
i14 is launching a weekly online discussion group for researchers exploring the intersection of generative AI, mathematics, and cognitive science. We analyze how architectural design impacts learning, memory, and reasoning in foundation models.
Join us to dissect training dynamics and explore how cognitive principles can inform the next generation of architectures.
Our next session will be hosted via Google Meet on: Sunday, April 13 · 12:00 PM AEST (Melbourne) which is Sunday, April 13 · 10:00 AM CST (Shanghai) which is Saturday, April 12 · 7:00 PM PDT (San Francisco)
Apply to join HERE: i14.ai/journal-club/