
Large Model Systems Organization: We developed SGLang @sgl_project (https://teapawt.net/t.co/OjwQadINKU), Chatbot Arena (now @arena), and Vicuna!
US
Joined August 2024
- Tweets1.4K
- Following205
- Followers18.3K
- Likes1.5K
Introducing the SGLang Consortium at SGLang Summit.
@nvidia , @AMD, @Google, @Meta, @intel, @Cloudflare, @modal, and @radixark are joining SGLang’s maintainers to help shape the project’s roadmap and advance open-source AI inference.
@sgl_project runs on hundreds of thousands of GPUs and powers some of the largest production inference deployments in the world.
The consortium will turn production needs into shared engineering work: supporting frontier models, improving reasoning and multimodal workloads, scaling distributed inference, and optimizing across hardware platforms.
Optimizing DeepSeek-V4.1-Flash PD & Colocate Serving on H20-3e
The AI Infra team at @Ant_Intl optimized DeepSeek-V4.1-Flash on 8×H20-3e with PD disaggregation. The blog stacks seven changes one at a time and measures each.
Highlights:
• Mean E2E 13.63 s → 4.70 s, 2.91× output throughput vs TP4 (64K-token prompts)
• TTFT 12.04 s → 1.89 s, at the cost of TPOT 3.20 → 5.62 ms
• One 4-GPU colocated worker: E2E 18.62 s → 7.96 s
Read the full blog below ⬇️
Full blog: lmsys.org/blog/2026-11-19-de…
🚀 Humming in SGLang: Flexible Low-Bit Kernels for Efficient MoE Serving
Humming JIT-compiles low-bit kernels for dense and MoE layers on NVIDIA GPUs from Turing to Blackwell. On Hopper, its FP8 path keeps expert weights packed at 4 bits and reconstructs FP8 operands in registers just before the Tensor Core multiply.
We tested @deepseek_ai’s DeepSeek-V4.1-Flash-1017 on 8 × H20-3e GPUs (TP8/EP1), keeping dense layers on their native FP8 path.
- Humming vs. Marlin: With MXFP4 weights and BF16 expert activations on both backends, Humming delivered a 1.07-1.23× prefill speedup on 4K–1016K prompts, measured by time to first token.
- FP8 vs. BF16 within Humming: On the same checkpoint, switching expert activations to FP8 increased output throughput by 17-34% at 1, 32, and 64 concurrent requests.
Read the full post from the Venus and SCT teams at @AntGroup and the @sgl_project community 👇
Full blog: lmsys.org/blog/2026-11-18-hu…
2
407
Optimizing DeepSeek-V4.1 on H200 with SGLang PD Disaggregation
The AI Infra team at @Ant_Intl served DeepSeek-V4.1-Flash from @deepseek_ai on 8×H200 with PD disaggregation.
This blog walks through how they tackled it: a cheaper sparse-attention Indexer, bigger Prefill chunks, leaner MoE communication, and late layers that compute only the last 128 tokens.
Highlights:
· E2E 15%–64% lower, TTFT 41%–79% lower vs the original baseline, across 8 cases (64K in / 512 out, 50/70/90% prefix hit, concurrency 4–32)
· The 128-token mode cuts TTFT by up to a further 46.7%
· The cost: TPOT 3%–52% higher, GSM8K 96.74% → 95.38%
Read the full blog below
It was so good to meet everybody and share these lovely memories with the entire community! See you next year!
And thanks for Tao Lin (@nblintao) for the excellent write-up!
teapawt.net/nblintao/status/212131…
I feel very fortunate to have had early access to @NVIDIAAI’s Vera CPU servers and the opportunity to run agentic RL with @radixark Miles, with every sandbox on Vera.
I wrote up our findings here. To our knowledge, these are the first public end-to-end agentic RL results on Vera CPU servers.
Huge thanks to the NVIDIA team and our own RadixArk team for making this possible!
The full announcement: lmsys.org/blog/2026-11-12-sg…
Day 1 at #SGLangSummit going strong! Just now on the Main Stage:
• Ying Sheng (@ying11231) announces SGLang v1.0: faster inference, better memory use, & flexible scaling for prod
• Banghua Zhu (@BanghuaZ) introduces Miles v0.2: Production RL across more hardware and workloads
🚀 Unified Memory in SGLang: One Arena for Every Token Lifetime
Hybrid models mix full-attention KV, sliding-window KV, and recurrent state. Fixed memory partitions waste GPU capacity. We built a shared arena where memory dynamically follows demand.
Key innovations:
🔹 No partitioning knobs, any page size, zero cross-type fragmentation
🔹 Token-centric KV layout, including speculative draft KV
🔹 Smart admission that rearranges memory to fit new requests
🔹 Safe asynchronous page reuse during in-flight GPU work
Read the full blog 👇
Unified Memory Roadmap: github.com/sgl-project/sglan…
Muse Spark Open: 517 → 1,109 tokens/s on 8× GB300 with SGLang megakernels.
One kernel launch per layer, overlapping compute, memory loads and communication. Total launches per decode step drop from 1,450 to 97.
lmsys.org/blog/2026-11-05-mu…
In our new technical post, Pranjal Shankhdhar explains what makes this work:
- Prefetching weights into shared memory and L2 before they’re needed.
- Warming the instruction cache so large kernels don’t stall fetching their own code.
- Using data itself as a readiness signal for cross-GPU communication.
- Fusing the draft model to make drafting 2.6× faster.
The post also includes measured timelines, CUDA/PTX examples, and the tradeoffs behind stopping at one megakernel per layer.










