LMSYS Org (@lmsysorg) profile picture

Large Model Systems Organization: We developed SGLang @sgl_project (https://teapawt.net/t.co/OjwQadINKU), Chatbot Arena (now @arena), and Vicuna!

US
Joined August 2024
Photo by LMSYS Org (@lmsysorg)Photo by LMSYS Org (@lmsysorg)Photo by LMSYS Org (@lmsysorg)Photo by LMSYS Org (@lmsysorg)Photo by LMSYS Org (@lmsysorg)Photo by LMSYS Org (@lmsysorg)Photo by LMSYS Org (@lmsysorg)Photo by LMSYS Org (@lmsysorg)Photo by LMSYS Org (@lmsysorg)Photo by LMSYS Org (@lmsysorg)
Introducing the SGLang Consortium at SGLang Summit. @nvidia , @AMD, @Google, @Meta, @intel, @Cloudflare, @modal, and @radixark are joining SGLang’s maintainers to help shape the project’s roadmap and advance open-source AI inference. @sgl_project runs on hundreds of thousands of GPUs and powers some of the largest production inference deployments in the world. The consortium will turn production needs into shared engineering work: supporting frontier models, improving reasoning and multimodal workloads, scaling distributed inference, and optimizing across hardware platforms.
30
18
11
104
18,237
Optimizing DeepSeek-V4.1-Flash PD & Colocate Serving on H20-3e The AI Infra team at @Ant_Intl optimized DeepSeek-V4.1-Flash on 8×H20-3e with PD disaggregation. The blog stacks seven changes one at a time and measures each. Highlights: • Mean E2E 13.63 s → 4.70 s, 2.91× output throughput vs TP4 (64K-token prompts) • TTFT 12.04 s → 1.89 s, at the cost of TPOT 3.20 → 5.62 ms • One 4-GPU colocated worker: E2E 18.62 s → 7.96 s Read the full blog below ⬇️
8
11
3
35
2,882
🚀 Humming in SGLang: Flexible Low-Bit Kernels for Efficient MoE Serving Humming JIT-compiles low-bit kernels for dense and MoE layers on NVIDIA GPUs from Turing to Blackwell. On Hopper, its FP8 path keeps expert weights packed at 4 bits and reconstructs FP8 operands in registers just before the Tensor Core multiply. We tested @deepseek_ai’s DeepSeek-V4.1-Flash-1017 on 8 × H20-3e GPUs (TP8/EP1), keeping dense layers on their native FP8 path. - Humming vs. Marlin: With MXFP4 weights and BF16 expert activations on both backends, Humming delivered a 1.07-1.23× prefill speedup on 4K–1016K prompts, measured by time to first token. - FP8 vs. BF16 within Humming: On the same checkpoint, switching expert activations to FP8 increased output throughput by 17-34% at 1, 32, and 64 concurrent requests. Read the full post from the Venus and SCT teams at @AntGroup and the @sgl_project community 👇
10
1
38
2,284
Optimizing DeepSeek-V4.1 on H200 with SGLang PD Disaggregation The AI Infra team at @Ant_Intl served DeepSeek-V4.1-Flash from @deepseek_ai on 8×H200 with PD disaggregation. This blog walks through how they tackled it: a cheaper sparse-attention Indexer, bigger Prefill chunks, leaner MoE communication, and late layers that compute only the last 128 tokens. Highlights: · E2E 15%–64% lower, TTFT 41%–79% lower vs the original baseline, across 8 cases (64K in / 512 out, 50/70/90% prefix hit, concurrency 4–32) · The 128-token mode cuts TTFT by up to a further 46.7% · The cost: TPOT 3%–52% higher, GSM8K 96.74% → 95.38% Read the full blog below
9
6
2
61
3,775
It was so good to meet everybody and share these lovely memories with the entire community! See you next year!
That's a wrap on SGLang Summit 2026! Incredible energy all around: the main stage, the expo hall, the booths, the workshops, and all the conversations that happen around them! To everyone in the community: THANK YOU for the wonderful memories! You make SGLang better 🧡🧡🧡
1
1
15
2,100
Faster GPUs make CPU speed matter more in agentic RL. In Miles tests under load, @nvidia Vera CPUs cut average episode time by 11%. Less waiting for tools means samples return sooner. Results + methodology: lmsys.org/blog/2026-11-13-mi… More details below🧵
9
2
2
54
7,342
Across the broader project, Miles ran 2,400+ episodes with every sandbox on Vera, in synchronous and fully asynchronous RL. Thanks to @NVIDIAAI for the collaboration!
1
5
479
And thanks for Tao Lin (@nblintao) for the excellent write-up! teapawt.net/nblintao/status/212131…
I feel very fortunate to have had early access to @NVIDIAAI’s Vera CPU servers and the opportunity to run agentic RL with @radixark Miles, with every sandbox on Vera. I wrote up our findings here. To our knowledge, these are the first public end-to-end agentic RL results on Vera CPU servers. Huge thanks to the NVIDIA team and our own RadixArk team for making this possible!
1
4
391
Day 1 at #SGLangSummit going strong! Just now on the Main Stage: • Ying Sheng (@ying11231) announces SGLang v1.0: faster inference, better memory use, & flexible scaling for prod • Banghua Zhu (@BanghuaZ) introduces Miles v0.2: Production RL across more hardware and workloads
5
6
36
4,013
🚀 Unified Memory in SGLang: One Arena for Every Token Lifetime Hybrid models mix full-attention KV, sliding-window KV, and recurrent state. Fixed memory partitions waste GPU capacity. We built a shared arena where memory dynamically follows demand. Key innovations: 🔹 No partitioning knobs, any page size, zero cross-type fragmentation 🔹 Token-centric KV layout, including speculative draft KV 🔹 Smart admission that rearranges memory to fit new requests 🔹 Safe asynchronous page reuse during in-flight GPU work Read the full blog 👇
10
10
3
113
14,499
Muse Spark Open: 517 → 1,109 tokens/s on 8× GB300 with SGLang megakernels. One kernel launch per layer, overlapping compute, memory loads and communication. Total launches per decode step drop from 1,450 to 97. lmsys.org/blog/2026-11-05-mu… In our new technical post, Pranjal Shankhdhar explains what makes this work: - Prefetching weights into shared memory and L2 before they’re needed. - Warming the instruction cache so large kernels don’t stall fetching their own code. - Using data itself as a readiness signal for cross-GPU communication. - Fusing the draft model to make drafting 2.6× faster. The post also includes measured timelines, CUDA/PTX examples, and the tradeoffs behind stopping at one megakernel per layer.
9
10
9
101
30,979