
CS PhD student @PrincetonCS, Currently Intern @nvidia, Previously CS MEng + BS @CornellCIS
Joined November 2021
- Tweets80
- Following202
- Followers1.1K
- Likes166
[6/N]
All SonicMoE Grouped GEMM kernels are built on QuACK. Each kernel overrides a single function. The most complex kernel (dH backward) only adds ~200 LoC for SonicMoE on top of QuACK, and the same code runs on Hopper and Blackwell GPUs.
[5/N]
On Hopper, we leverage Ping-Pong warpgroup scheduling to overlap the heavy epilogue IO with the tiled GEMM computation.
On Blackwell, the hardware does it differently but in the same spirit: we have a dedicated on-chip tensor memory (TMEM) split into two accumulator stages. The MMA warp fills one stage while epilogue warps handle the other stage, then they swap. The heavy epilogue IO is overlapped with the tiled GEMM again, which we elaborate on in the blogpost.
[4/N]
SonicMoE is designed for fine-grained MoEs. It achieves greater relative speedup over existing MoE baselines when we increase the expert granularity on B300 GPUs.
[3/N]
On B300 GPUs, SonicMoE achieves 54% & 35% higher fwd/bwd TFLOPS than the DeepGEMM baseline and 21% higher fwd TFLOPS than the triton official example across 6 open-source MoE configs (7B to 685B). SonicMoE often doubles the achieved TFLOPS over ScatterMoE and MoMoE.
[2/N]
Modern MoEs are scaled towards the fine-grained regime where we have more smaller experts to activate. However, the activation memory footprint of existing MoE kernels will linearly increase and drain the VRAM resources. We instead compute the MoE backward pass in a different but math-equivalent way. Activation memory usage is now the same as a dense model with equal activated parameters, with 45% reduction from community MoE kernels. SonicMoE’s activation memory usage is also independent of expert granularity.
🚀SonicMoE🚀now runs at peak throughput on NVIDIA Blackwell GPUs 😃
54% & 35% higher fwd/bwd TFLOPS than the DeepGEMM baseline and 21% higher fwd TFLOPS than the triton official example. SonicMoE still maintains its minimum activation memory footprint: the same as a dense model with equal activated parameters and independent of expert granularity. We wrote a blogpost on how we leveraged Blackwell features and the software abstraction on QuACK:
Work with @MayankMish98, @XinleC295, @istoica05, @tri_dao
[5/N]
The token rounding routing eliminates padding waste in grouped GEMM and can achieve 16% relative speedup over TC top-K on kernel computation time while delivering robust token-choice accuracy even under highly sparse MoE training regimes.
[3/N]
SonicMoE achieves 50%+ speedup compared to existing baselines for a single MoE layer with open-source MoE configs from 7B to 685B. For 7B MoE model with FSDP-2, SonicMoE on 64 H100s gets 213B tokens/day while ScatterMoE (previous SOTA) on 96 H100s gets 225B tokens/day.
[2/N]
The activation memory usage is same as a dense model with equal activated parameters (minimum activation memory required for backward computation without doing activation recomputation with GEMM). SonicMoE’s activation memory usage is also independent of expert granularity.
🚀SonicMoE🚀: a blazingly-fast MoE implementation optimized for NVIDIA Hopper GPUs. SonicMoE reduces activation memory by 45% and is 1.86x faster on H100 than previous SOTA😃
Paper: arxiv.org/abs/2512.14080
Work with @MayankMish98, @XinleC295, @istoica05, @tri_dao
[5/N]
H100 Feature: Cluster Reduction
This is fun bc it’s new and no one has really written a tutorial about this. 🤗 We did this in ~30 lines of python. 🤯 Gets SOL for large reduction dim (e.g. up to 262k) 🚀
[4/N]
Warp Reduction & Block reduction
After each thread reduces all values on its own registers, we will move down the memory hierarchy to registers owned by other threads (warp reduction), and use a small reduction buffer for local writes and loads in the thread block.
[3/N]
Efficient Load & Store
We explore how to asynchronously vectorize 🛒 the loads from global memory to shared memory, and to the registers. Finally, we will vectorize 🛒 store our results.
🦆🚀QuACK🦆🚀: new SOL mem-bound kernel library without a single line of CUDA C++ all straight in Python thanks to CuTe-DSL. On H100 with 3TB/s, it performs 33%-50% faster than highly optimized libraries like PyTorch's torch.compile and Liger. 🤯
With @tedzadouri and @tri_dao















