Kuntai Du (@this_will_echo) profile picture

Chief Scientist | Committer of vLLM / LMCache / Production Stack

Joined January 2022
Photo by Kuntai Du (@this_will_echo)Photo by Kuntai Du (@this_will_echo)Photo by Kuntai Du (@this_will_echo)Photo by Kuntai Du (@this_will_echo)Photo by Kuntai Du (@this_will_echo)Photo by Kuntai Du (@this_will_echo)Photo by Kuntai Du (@this_will_echo)Photo by Kuntai Du (@this_will_echo)Photo by Kuntai Du (@this_will_echo)Photo by Kuntai Du (@this_will_echo)
Jensen Huang is leaving NVIDIA. But he changed the history of computing. When others were on CPU, he bet on GPUs. When others were on gaming, he bet on CUDA. When others were on single-GPU perf, he bet on architecture. And I grow from a gamer to an AI infra developer.
79
Replying to @lihanc02
I guess ur talking about compression? It is not widely accepted in industry and I don't know why...
1
75
Replying to @woongworks
Existing systems mainly suffer from writing things that are unlikely to be reused
1
28
We posted our paper on how fast LLM prompt caching can wear out SSDs on arXiv. Literally 6 hours later, another paper on the exact same topic dropped. The competition is sooooo real 💀 Their paper (SARIA): arxiv.org/abs/2610.31169 Our paper (Gandalf): arxiv.org/abs/2610.30791
2
20
1,737
Kuntai Du retweeted
Today we’ve announced to the world that 𝐓𝐞𝐧𝐬𝐨𝐫𝐦𝐞𝐬𝐡 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦, the 𝐦𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭 𝐥𝐚𝐲𝐞𝐫 𝐟𝐨𝐫 𝐀𝐈 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐝𝐚𝐭𝐚, is officially live and enterprise-ready 🚀 Read the full announcement 👇 na2.hubs.ly/H08nsMN0
4
1
12
289
Kuntai Du retweeted
Day 1 of The AI Conference SF is in the books! 🎉 Come say hi to the team at Booth #214. We're excited to show how Tensormesh Platform can speed up AI inference and cut GPU costs with intelligent KV caching. See you on Day 2! #TheAIConference #AIInfrastructure #LLMInference #KVCache
1
4
8
200
Besides just KV cache, what else can we do to make LLM agents faster?🧐 Check our (LMCache team) latest token dropping API! Dev-facing, easy-to-use. Drop half of tokens, ur LLM agent will then decode 1.6x faster! Jupyter notebook example: github.com/LMCache/LMCache/b…
1
1
11
1,340
Replying to @lmcache
5. Looking back, we success because of solid engineering AND the right vision brought by research. We bet on the right direction, got there early, and kept building. So when the trend of KV cache hits, we are ready.
15
Replying to @lmcache
5. After LMCache got wide adoption, we received tons of feedback. The main one is that: we found deployment was too tightly coupled with vLLM. So we decided to separate LMCache from engine process.
18
Replying to @lmcache
4. The real turning point came in Q2–Q3 2025. Agentic workloads outgrew GPU memory, industry started reaching out, and more open-source contributors joined LMCache.
14
Replying to @lmcache
3. Later, due to a major vLLM upgrade (vLLM v1), Yihua and I redesigned the connector API. That API is still used today 🎉
28
Replying to @lmcache
2. Our first version patched vLLM. Very painful to maintain. I was in vLLM core team those days and realized that LMCache and PD disaggregation were all doing KV cache I/O. So I wrote the first vLLM KV cache connector API. After that LMCache needs no patch.
37
Replying to @lmcache
1. We started in Prof. Jiang's group at the University of Chicago. When LLMs took off, we felt that it will be the new "CPU" for future computation, and KV cache will be the new big data. So Yihua started LMCache in July 2024, a project just for KV cache.
46
LMCache just hit 10,000 GitHub stars!🎉 2 years, we grow from a toy lab project to an open-source project used by industry and mentioned in NVIDIA GTC slides. As one of the early contributors and committers, here's my view of story 🧵
𝐋𝐌𝐂𝐚𝐜𝐡𝐞 𝐣𝐮𝐬𝐭 𝐡𝐢𝐭 𝟏𝟎,𝟎𝟎𝟎+ 𝐆𝐢𝐭𝐇𝐮𝐛 𝐬𝐭𝐚𝐫𝐬! ⭐ Thank you to every contributor, user, researcher, and engineer who has helped build, test, improve, and share LMCache. This milestone belongs to everyone who's been part of the community, and we're just getting started! In just the past year, we've: ⚡ Shipped multiprocess (MP) mode, reducing time-to-first-token by up to 13× on MoE models by decoupling the KV cache from the inference engine 🔗 Introduced peer-to-peer KV cache transfer, enabling instances to fetch cached KV directly from one another instead of recomputing it ⚙️ Expanded inference engine support to TensorRT-LLM and SGLang 💾 Opened the storage adapter framework, making it easy to build and ship new storage backends without modifying the LMCache core 🖥️ Added multi-platform dev support, so you can build and test LMCache without needing a GPU 𝐑𝐞𝐚𝐝𝐲 𝐭𝐨 𝐡𝐞𝐥𝐩 𝐛𝐮𝐢𝐥𝐝 𝐰𝐡𝐚𝐭'𝐬 𝐧𝐞𝐱𝐭? Whether you're looking for your first open-source contribution, already using LMCache, or interested in AI infrastructure, there are plenty of ways to get involved: 💬 Join our Slack → join.slack.com/t/lmcachework… 🛠️ Find an issue to work on → github.com/LMCache/LMCache/i… 🌐 Explore and star the project → github.com/LMCache/LMCache Here's to the next 10K! 🎉 #LMCache #LLMInference #AIInfra #OpenSource #KVCache
7
1,321
I was delighted to give three keynotes recently, one at the 3rd HotInfra workshop at ISCA'26 (hosted by Prof. Jian Huang), one at the UIUC Systems Groups retreat (hosted by @tianyin_xu ), and a CMU guest lecture (hosted by Prof. Lei Li). The message was simple: KV cache is BEYOND cacheable intermediate states. It is what makes inference systems SMART. - If cached, it saves cost and reduces latency. - If analyzed, it reveals how the model attends to and understands context. - If optimized, it can improve not only performance, but also output quality. It's not a liability, but an ASSET --- (almost) always the more you have, the better. That is the part most people in industry still underestimate, even though the techniques are already here today: KV cache compression, blending, semantic reuse, cross-model sharing, attention steering, KV-cache-based retrieval, and more. The gap is not imagination. The gap is productization. KV cache will become a core AI-native data layer in future inference infrastructure — not just something we store, but something we manage, analyze, and optimize. #KVCache, #LMCache, #LLMInference, #Tensormesh, #vLLM, #SGLang, #MLSys, #HotInfra
2
5
2
69
6,961