Stanford (MS '26 - ECE) - [email protected] UT Austin (BS '24 - CS (Turing), ECE, Math)
SF Bay Area
Joined December 2023
- Tweets1
- Following46
- Followers15
- Likes5
Context-Sharded Block Parallelism (CSBP) speeds up long-context Diffusion LLMs via clean context sharding & block load balancing—up to 7.59× speedups! 🚀
Open-sourced in Turbo-dLLM.
W/ @TarunSures41845 @hangoo_kang @PranshuChatur11 @ishanskhare @KumbongHermann @Azaliamirh
Diffusion LLMs and speculative decoding promise much faster agents. Yet agents need long contexts, and training on them is painfully slow.
Introducing Context-Sharded Block Parallelism (CSBP), a new distributed parallelism strategy unlocking significant training efficiency for diffusion LLMs, with speedup gains growing with context length 🚀
⚡ 7.59× faster DFlash2 speculative decoding drafter training
⚡ 1.61× faster block diffusion fine-tuning
⚡ 1.33× faster autoregressive → block diffusion adaptation
With the same GPU hours, models trained with CSBP score higher on SWE-bench Verified and Terminal-Bench Lite 👑
Open-sourced in Turbo-dLLM, our new optimized distributed training library.
Advised by @Azaliamirh and with an amazing team: @PranshuChatur11 @hangoo_kang @pshroff_ @ishanskhare @KumbongHermann
