Adam Louly (@LoulyAdam) header
Adam Louly (@LoulyAdam) profile picture

Inference Engineering @Microsoft

Bellevue, WA
Joined April 2017
Photo by Adam Louly (@LoulyAdam)Photo by Adam Louly (@LoulyAdam)Photo by Adam Louly (@LoulyAdam)Photo by Adam Louly (@LoulyAdam)Photo by Adam Louly (@LoulyAdam)Photo by Adam Louly (@LoulyAdam)Photo by Adam Louly (@LoulyAdam)Photo by Adam Louly (@LoulyAdam)Photo by Adam Louly (@LoulyAdam)Photo by Adam Louly (@LoulyAdam)
What are we going to do with millions of H100s, A100s, and older gen GPUs as newer hardware takes over? By my estimates, a B200 can recover its price premium over a used H100 in just ~77 days at 75% utilization. Has anyone found a genuinely cost effective way to keep older GPUs competitive?
1
1
1
618
1T params model running on 1bit has the same quality as a 100b params model on full precision, Its pointless 🤷‍♂️
🟠 RESULTS: I ran a 1 trillion parameter model on my 64GB M5 Pro Mac mini. Le Chonk by @MistralAI (Mistral Large 4, 1T total, 52B active), using @UnslothAI’s 1-bit quant. That’s 233GB of weights on a machine with 64GB of memory, so it streamed straight off the SSD. I guessed 2-4 tok/s. I was wrong. - 0.35 tok/s decode, about 2.9 seconds per token - 3.0 tok/s prefill - ~57GB peak memory - 3 runs, all within 0.01 tok/s of each other The answers were coherent. It summarized a doc correctly in two sentences. It’s too slow for chat on a Mac mini. It’s also a trillion parameter model running on my desk with no API key. Only open weights make that possible.
1
174
I was today's years old when I discovered you can actually use a collector_op on tcgenn05_mma to cache A tile on a dedicated buffer .. I'm curious how many custom blackwell kernels actually take advantage of this, personally I haven't seen much. cutlass docs explains is much better than cuda ptx isa docs.
3
158
Yet another misleading benchmark (batch size of 1).
At 3,000 tokens/s, your budget per token is 333 microseconds. A basic kernel launch is 4.5µs. Kog hit 3k tok/s on standard GPUs by dropping PyTorch and writing a persistent monokernel in raw assembly. The CPU scheduler is officially the enemy ✨
2
414
Out of all the MLSYS happy hours, OpenAI’s one was probably the top 1, together AI’s was also dope!
4
271
I’m at MLSys this week! feel free to come say hi if you want to chat about inference, GPU programming or ml systems
7
242
Unrolling a single mfma loop produced 17% perf gain 🤯🤯 I’ll never underestimate the power of unrolling ever again
1
3
287
Agentic coding is like makeup, a lot of engineers started believing they’re naturally that good.
5
142
Im glad my feed no longer shows these cringe meta expressions such as: Ngmi Lfggg We’re so back We’re cooked And the list goes on..
105
Guess I should start an FPL twitter account for fpl tips
1
149
Its more interesting when you know what he’s referring to at the end
Ilya Sutskever just told the AI industry why scaling is finished. One word built it. One word is about to break it. Sutskever: “Scaling is just one word, but it’s such a powerful word because it informs people what to do.” For five years, that single word replaced an entire research culture. Nobody needed breakthroughs. They needed bigger checks. Sutskever: “If you mix some compute with some data into a neural net of a certain size, you will get results, and you will know that it will be better if you just scale the recipe up.” That’s not science. That’s a recipe. Sutskever: “Companies love this because it gives you a very low risk way of investing your resources.” The most transformative technology in human history ran on the same logic used to franchise a restaurant chain. More locations. More ingredients. Same recipe. Predictable returns. You didn’t need researchers who could see around corners. You needed accountants who could approve purchase orders. But recipes expire. Sutskever: “At some point though, pre-training will run out of data. The data is very clearly finite.” Five years of infrastructure. Five years of hiring. Five years of investor decks. All built on top of something temporary. Sutskever: “I don’t think that’s true.” The co-founder of OpenAI. The mind behind the breakthroughs that made this entire era possible. Saying more money won’t solve it. Sutskever: “In some sense we are back to the age of research.” Most of the companies racing to build AGI were never research companies. They were scaling companies. They hired for execution. Not discovery. They optimized for throughput. Not insight. The talent pipelines. The investor pitches. The board decks. All built around one assumption. That the recipe would never expire. It’s expiring. And the companies that spent five years perfecting the art of spending money are about to discover something. The next era demands what capital can’t purchase. An original idea.
2
162
Highly recommended ✅
This quoted post is unavailable.
200
There should be a standard for layout contracts, repacking wgmma optimzied layouts to mfma optimized layouts is a nightmare. It would be ideal if there’s a standard to at least repack every layout to a row major one then, it would also be cool to have a portable decode function with different layouts for the same hardware and tune against…
101
In 6 months, all cuda/hip dsls, as well as frameworks like triton and gluon, will primarily be used as hardware abstraction layers for cross-vendor and custom-chip support. Most highly optimized kernels will instead be written directly in low level cuda and hip. In 1 year, llms will be able to one shot instruction level kernels of custom silicon by ingesting the docs in the context.
5
118
My team at Microsoft is still hiring inference engineers, we have positions from mid to principal levels. You’ll get to work directly on OpenAI models (gpt 5.4+) and optimize them for different workloads. You’ll get to work on the latest hardware out there, GB300s, H200s, MI300X+ Dm me for direct referral
4
296
AI bill for this week ... this is getting out of hand.
3
3
829
How do you entertain yourself while waiting for your codex-5.4-X-Mega-High-Thinking to finish its task?
3
373
Is it ever legal to leave a printf in a cuda kernel if it resolves a heisenbug?
1
2
305
Is there a way to have one agent act as an orchestrator so you only talk to that agent, and it then manages the other 4-5 agents, gives you status updates on each, and dispatches tasks to them?
1
2
261