Junchen Jiang (@JunchenJiang) header
Junchen Jiang (@JunchenJiang) profile picture

CEO @Tensormesh, CS Prof @ UChicago https://teapawt.net/t.co/PIHealkTNi https://teapawt.net/t.co/hoetjwXKIt

Chicago, IL
Joined September 2012
Photo by Junchen Jiang (@JunchenJiang)Photo by Junchen Jiang (@JunchenJiang)Photo by Junchen Jiang (@JunchenJiang)Photo by Junchen Jiang (@JunchenJiang)Photo by Junchen Jiang (@JunchenJiang)Photo by Junchen Jiang (@JunchenJiang)Photo by Junchen Jiang (@JunchenJiang)
Junchen Jiang retweeted
When a vLLM worker on an Ascend NPU restarts, does it have to lose its warm KV cache? With LMCache multi-process (MP) mode, now supported on Ascend NPUs, the cache stays. A standalone lmcache server owns the KV cache, so the cache survives engine restarts, and several vLLM instances share it. The harder question is how KV tensors cross the process boundary on an NPU. The new post explains the two native transfer paths, why DeepSeek-V4 works with no extra configuration, and how to start in three steps. βœ… Supported on Atlas 800I A2 and A3 and validated with Qwen models and DeepSeek-V4. The Ascend C kernels are open source. na2.hubs.ly/H091L0N0 #LMCache #KVCache
2
3
21
968
Junchen Jiang retweeted
Running LMCache across a fleet? Each KV cache server only knows about itself, so the same prompt gets prefilled again on every node. The LMCache Coordinator adds a control plane on top: 𝐨𝐧𝐞 πŠπ• 𝐜𝐚𝐜𝐑𝐞 𝐟𝐨𝐫 𝐭𝐑𝐞 𝐰𝐑𝐨π₯𝐞 𝐟π₯𝐞𝐞𝐭. It’s the cross-node layer in Tensormesh Platform, our self-hosted KV caching for vLLM on Kubernetes πŸ‘‡ na2.hubs.ly/H08-ygm0
3
5
199
Junchen Jiang retweeted
When an RL job spreads sixteen samples of one prompt over four nodes, why does each node run prefill for that same prompt again? In LMCache multi-process (MP) mode, each node’s KV cache server sees only itself. Node B does not know that node A already holds the KV. ✨ LMCache contributor Rui Zhang introduces the LMCache coordinator: a control plane that builds a fleet-wide view of every server’s cache and lets pluggable policies act on it. It is opt-in, in LMCache v0.5.6 and later, and stays out of the request path. How it stays correct through restarts, how to add your own policy, and where it goes next: blog.lmcache.ai/en/2026/11/1… #LMCache #KVCache
1
3
1
22
899
Junchen Jiang retweeted
Where should the KV cache be stored, and what does it cost to move it? At KVCache Meetup Korea on September 30, speakers from XCENA, SK hynix, FuriosaAI, and LMCache answered from different layers: CXL memory pools, GPU and HBM bandwidth, request scheduling on RNGD, and LMCache MP Mode. PyTorchKR and XCENA co-hosted, and Tensormesh sponsored. Junghwan Park of PyTorchKR wrote the recap in English and Korean. It includes a test where retrieving a host cache was faster than recomputing it, and what attendees want at the next meetup. na2.hubs.ly/H08Qmxk0 #LMCache #KVCache
3
16
624
Junchen Jiang retweeted
Our first LMCache meetup in the San Francisco Bay Area was on 22 October 2026 at Samsung Semiconductor in San Jose, co-hosted by Dongjoo Seo of Samsung and Karsten Wade of Tensormesh. Samuel Shen of Tensormesh led a hands-on notebook session. Luis Chamberlain of Samsung spoke on asymmetric quantization, dma-buf with io_uring, and Cartridges. Dongjoo Seo closed with LMCache on CXL memory. If you missed it, the recap has all the slides and Sam's notebook, which runs LMCache on a laptop with no GPU. na2.hubs.ly/H08J0bq0
1
4
10
543
Junchen Jiang retweeted
inference engines should be bending over backwards more to get cache hits really interesting direction: > An Internet for the KV Cache - sending a KV cache across the Internet can now be faster and cheaper than computing it again.
20
18
5
331
30,081
Junchen Jiang retweeted
When is it better to fetch a KV cache than to recompute it? New on the LMCache blog: an Internet for the KV cache, by Siddhant Ray, adapted from his arXiv paper with Nick Feamster and Junchen Jiang of the University of Chicago. One example: DeepSeek-V4-Pro, about 100,000 tokens, about 1 GB of compressed KV cache. Sending it over a 10 Gbps link takes about 0.8 s and $0.09. Recomputing it on AWS 8Γ—B200 takes about 15 s and $0.47. Under the post’s assumptions, transfer is about 19Γ— faster and 5.2Γ— cheaper ⚑ (AWS prices are examples only.) blog.lmcache.ai/en/2026/10/2…
11
14
7
147
12,622
Junchen Jiang retweeted
Today we’ve announced to the world that π“πžπ§π¬π¨π«π¦πžπ¬π‘ 𝐏π₯𝐚𝐭𝐟𝐨𝐫𝐦, the 𝐦𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭 π₯𝐚𝐲𝐞𝐫 𝐟𝐨𝐫 π€πˆ 𝐒𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐝𝐚𝐭𝐚, is officially live and enterprise-ready πŸš€ Read the full announcement πŸ‘‡ na2.hubs.ly/H08nsMN0
4
1
12
289
Junchen Jiang retweeted
πŸ“… New date: the LMCache hands-on workshop moves to Thursday 22 October. Same time, 1:30 to 5:30 PM Pacific. Same place, Samsung, 3655 N First St, San Jose. Still free, and registration is required. Four hours with a live LMCache environment, engineering sessions, and a reception to close. It falls during Open Source AI Week in the Bay Area. Bring a laptop for the hands-on part. Know someone who could not attend on 6 October? Send this to them. πŸ” na2.hubs.ly/H08k5bM0
1
3
188
Junchen Jiang retweeted
We're looking fsorward to being there. If you drop by our booth, we've got a snappy new LMCache ball cap for you. (And we'd love you to star our repo at github.com/LMCache/LMCache and follow us @LMCache!)
We’re proud to announce the first Partners & Exhibitors joining ODSC AI West 2026: @googlecloud | @Microsoft | @databricks | @intel | @qdrant_engine | Starburst | @runpod | Cranium | Sphinx | @datasaurai | @lmcache | Okareo | @senzing | @jetbrains | Merge Dev | Squadbase From cloud and data infrastructure to vector search, inference, AI security, evaluation, and developer tools, these teams are helping organizations move AI systems from experimentation into production. At ODSC AI West, you can explore their technologies, see practical solutions in action, ask technical questions, and connect directly with the people building the AI ecosystem. With 2,500+ AI builders and leaders gathering this October, the conversations shaping your next project, platform decision, or technical strategy could happen here. πŸ”₯Last Chance -30% Off πŸ‘‰ Register now: hubs.li/Q04zzvc90 πŸ“ San Francisco πŸ—“οΈ October 27–29, 2026 | In-Person + Virtual #ODSCAI #AIWest #AIConference #AIInfrastructure #AIEngineering #DataScience #MachineLearning #AgenticAI
1
6
342
Junchen Jiang retweeted
𝑨𝑰 𝒄𝒐𝒔𝒕𝒔 𝒅𝒐𝒏'𝒕 𝒔𝒕𝒐𝒑 𝒂𝒕 𝒕𝒉𝒆 π’•π’π’Œπ’†π’. As models consume more tokens to tackle increasingly complex tasks, the compute, memory, storage, and energy behind them matter too. AI economics and π’Šπ’π’‡π’“π’‚π’”π’•π’“π’–π’„π’•π’–π’“π’† π’†π’‡π’‡π’Šπ’„π’Šπ’†π’π’„π’š are becoming the same conversation. ⬇️ na2.hubs.ly/H089px80
1
3
159
Junchen Jiang retweeted
Day 1 of The AI Conference SF is in the books! πŸŽ‰ Come say hi to the team at Booth #214. We're excited to show how Tensormesh Platform can speed up AI inference and cut GPU costs with intelligent KV caching. See you on Day 2! #TheAIConference #AIInfrastructure #LLMInference #KVCache
1
4
8
200
Junchen Jiang retweeted
The lmcache package on PyPI has passed 1,000,000 all-time downloads, per pepy.tech (CI traffic included). It stood at 1,002,795 on 2026-09-24. LMCache is an LLM serving engine extension that reduces time-to-first-token (TTFT) and increases throughput, especially in long-context scenarios. Current release: 0.5.5. Right now LMCache is a popular way to save money and reduce resource use for GPUs. If this doesn't matter to you directly, it probably matters to your customers. On GitHub, the project has 350+ contributors and about 11.9k stars. Thank you to everyone who has sent a pull request, reported an issue, or reviewed code. ⭐ github.com/LMCache/LMCache
3
1
6
435
Junchen Jiang retweeted
LMCache can now own an unmounted NVMe device as a KV cache tier: fixed-size slots, direct I/O, and checkpoints on the device, so the key index comes back after a crash. Dongjoo Seo, Ankit Kumar, Sangyoon Kwon, Daegyu Han, and Nayeon Kim of Samsung built Raw Block PR by PR. Their write-up also shows where it loses: when the page cache fits, buffered files can still win. So when is owning the device worth the alignment work? Before you try it, dedicate the device. Raw Block may overwrite what's there. blog.lmcache.ai/en/2026/09/2… #KV_cache #LMCache #Samsung #NVMe
5
1
16
875
πŸŽ‰Exciting joint work between Amazon #SafeMaker and #LMCache!
Prefill and decode want opposite things from a GPU. Chunked prefill reduces the interference without removing it. On Amazon SageMaker HyperPod the two phases get separate GPU pools, and LMCache's PD backend moves the KV cache between them. Per-token latency then stops climbing with concurrency, where a colocated deployment degrades. Whether that pays off comes down to one question: can the KV cache move between pools faster than a decoder can recompute it? Xuan Lu, Vinay Arora, and Swapnil Palod at AWS, with Yihua Cheng at Tensormesh, measured it on Llama-3.3-70B-Instruct and wrote up the NIXL and EFA path that makes the answer yes. blog.lmcache.ai/en/2026/09/1…
1
10
980
Junchen Jiang retweeted
Our CTO @YihuaCheng is on the technical panel at AI Infra Signal, Sep 12th in Sunnyvale. Topic: the future of AI infra, from scaling models to operating agents. KV cache layout, eviction, compression, cross-node sharing, tiering to DRAM/NVMe. 100 spots, application-only. Apply: na2.hubs.ly/H07JLDG0
1
2
159
Junchen Jiang retweeted
πŸ”¬Calling ML engineers: turn your KV-cache editing idea into a real benchmark with LMCache! KV-cache techniques such as token dropping, KV cache editing, or even cartridge-style KV cache training allows an LLM to decode much faster or produce much better text outputs. But how to easily integrate them into a production-level stack and get real benchmarking numbers? Here comes the LMCache SDK! πŸ’» With the LMCache SDK, you can finish all your implementation with changing just ONE cell of a Jupyter Notebook, and get real throughput and accuracy numbers in a real inference engine! πŸ”₯ Your KV-cache algorithm no longer has to stay inside a paper. blog.lmcache.ai/en/2026/09/0… Copy the notebook and try it out! colab.research.google.com/dr… Try the LMCache SDK: github.com/LMCache/LMCache/t…
4
1
12
958
Junchen Jiang retweeted
Tomorrow's LMCache Community Office Hour is an AMA. Karsten Wade (@quaid) has spent two months quietly reading the project β€” PRs, issues, Slack threads, plus an agentic scan. He'll share what he found, then answer anything you ask. Wed Sep 9, 11am PT lmcache-officehours.zapier.a…
1
2
251
Junchen Jiang retweeted
Kuntai Du is speaking at the @PyTorch Conference (Oct 20-21, San Jose) on scalable inference infra for agentic AI. We'll be there, come talk KV caching and vLLM serving with us. Register by 9/4: hubs.la/Q04tBgv_0 @this_will_echo
We’re bringing together open source leaders to tackle some of the biggest technical challenges in agentic AI at PyTorch Conference North America (Oct 20–21 in San Jose). Explore critical themes across the entire lifecycle, from adaptive post-deployment learning to vendor-neutral workload migration, scalable reinforcement learning infrastructure, high-throughput vLLM production serving and agent-assisted workflows for maintaining PyTorch itself. Featured Speakers include: @Google : Bill Jia @huggingface : Ben Burtenshaw, Sergio Paniego Blanco @IBM : Or Ozeri @intel : Eikan Wang, Huma Abidi @Meta : Yichuan Wang, Shuhua Yu, Felipe Mello, Jiani Wang, Edward Yang, Driss Guessous, Andrey Talman @adaption_ai : Sara Hooker @AgenticAIFdn : Mazin Gilbert, MBA, Ph.D. ByteDance: Neiwen LING @radixark : Mao Cheng @RedHat : Joseph Groenenboom, Tyler Michael Smith @tensormesh : Kuntai Du University of Virginia: Tianle Zhong Register for your standard ticket by September 4: hubs.la/Q04tBgv_0 + stay for AGNTCon + MCPCon that same week Read the Agentic AI and Next-Gen Sessions guide: pytorch.org/blog/agentic-ai-…
1
3
221