
CS PhD student @PrincetonCS, Currently Intern @nvidia, Previously CS MEng + BS @CornellCIS
Joined November 2021
- Tweets80
- Following202
- Followers1.1K
- Likes166
Wentao Guo retweeted
Dear researchers:
Did you know that your H100 can actually ray-trace like a gaming card? 🎮
for a long time, people have assumed that compute cards like the H100 can only compute, and that you need a graphics card like the RTX 5090 to render. we show that's wrong.
🚨Meet RTX H100.
Blender Cycles on the Open Data benchmark (Blender 5.3.0): ours 48,641. the RTX 5090 is at 19,202, and a stock H100 gets only 7,041. our original goal was to match the 5090. we ended up at 2.5x. our neural rendering approach is lossy. we test within a reasonable loss, and every result passes an image check.
we think nobody doing this before was an accident: graphics people rarely touch compute cards, and people who have compute cards rarely know graphics.
robotic simulators need fast rendering. now a general-purpose compute card can do it, so you don't need heterogeneous compute or a complicated cluster.
we'll extend this to more hardware next. almost ready for GTA 6 on the cluster 🤣
Best,
Wenhao
Wentao Guo retweeted
The era of one-size-fits-all inference engines is ending!
We had Claude write a separate megakernel for each of 8 open models, 8B - 594B: one GPU kernel that runs the entire decode loop.
Up to 1.77× faster than vLLM / SGLang, 2.05× with lossless model-specific tricks. 🚀
Wentao Guo retweeted
9/9 we're releasing nano-gpic: all 7 compute-optimal rungs from d10 to d18 (held-out loss 0.0679 to 0.0635, zero-CFG FD-DINOv2 950 to 325), one-command configs and eval tools.
larger rungs past 2.3e20 FLOPs are training now.
thanks to @Goodeat258, @zeng_boya, @HaochengXiUCB, @AngCao3, @SihanXuOfficial, @WentaoGuo7 and @EnxinSong for feedback and discussions 🙏
this task is added into OpenRSI Index @OpenRSI few months ago.
blog: wenhaochai.com/blogs/scaling…
code: github.com/wenhaochai/nano-g…
Wentao Guo retweeted
Spec decoding is lossless in distribution but fails exact match in precision 🫠, because in real-world serving, later draft slots get noisier.
We taught the drafter to fumble key decisions, where the target rejects it and writes them from the cleanest slot. +7.0 on DeepSWE for GLM-5.3, same speed 🚀.
Check our work led by @lhyTHU!
Speculative decoding has been purely treated as an acceleration mechanism. We found it's also an allocator for context precision. We trained 0.06% hosted parameters for deepseek-v4.1-flash-1017 and 0.28% for GLM-5.3 and significantly improved agentic coding: alphaxiv.org/abs/2611.advers…
Wentao Guo retweeted
🚀 Kernel Design Agents to Optimize Kimi Delta Attention (upto 2.96x)
When the KDA (Kernel Design Agents) project was initiated, its name happened to coincide with another KDA (Kimi Delta Attention). More than once, people asked us , "Can we use KDA to write a version of KDA?" Today, right at a moment with a full moon good for shooting, we are thrilled to share the latest results of KDA(gent)-v0.6: using KDA(gent) to optimize KDA(ttn)!
KDA-v0.6 achieves this through
* Humanize2 Flow & LLM Collaboration Optimization: Introducing multi-round iterative evolution with Flame Chase, GPT-5.6-Sol, and Fable-5, paired with an automated Workspace cleanup mechanism to significantly broaden the operator search space.
* Multilingual Domain-Specific Skill Stack Support: Fully supporting CuTe-DSL, CUDA C++, as well as recent Agent-Native CAKE IR and TIRx. Combined with customized diagnostic tools (such as CuTe's dedicated IKET pipeline analyzer and TIRx's CPU-side numerical simulation and static checks), achieving deep performance bottleneck diagnostics.
* Self-Evolving Kernel Wiki: Building a dynamically updated knowledge base that automatically cleans up invalid/erroneous implementations, corrects classification tags, and streamlines search indexing, significantly improving the Agent's knowledge retrieval quality and code generation accuracy.
KDAgent ultimately optimized the KDAttn kernel to 2.96x of FlashKDA, and we face a lot of unexpected hacks during the optimization. More details shared in our blogs nvlabs.github.io/kda/blog/20…
Release results of KDA for KDA: github.com/NVlabs/kda/tree/2…
Community & Production Deployment:
All kernels are open-sourced for community verification! For deploying NVIDIA Agentic CUDA acceleration in production environments, we recommend using the production-level version rigorously verified by FlashInfer.
By the Team: Dongyun Zou, Yixin Dong, Hongyi Jin, Junxian Guo, Yahui Cui, Avery Huang, Zihao Ye, Junru Shao, Changye Li, Zijian Zhang, Sihao Liu, Song Bian, Ligeng Zhu.
Wentao Guo retweeted
⚡ Attention kernels are well-optimized on GPU. But what about TPU?
🎯 We release Flywheel Attention: TPU-native softmax attention (MHA / GQA) + Gated DeltaNet kernels. Flywheel is 1.56× faster than the best open-source MHA/GQA kernel and 3.61× faster than the best open-source GDN kernel we could find. 👇
Wentao Guo retweeted
We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs.
No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase.
Meet Rigel 🧵
Wentao Guo retweeted
Open-source video generation is now faster than playback without compromising quality.
Introducing Video Delta Net (VDN): hybrid attention for live text-to-video with near-lossless quality.
VDN accelerates Minimax-H3 by 75 - 90 x, generating 14 seconds of 768p video in 11 seconds on 8× NVIDIA B200 GPUs.
Checkpoints + training/inference code + Technical Blog ⬇️
(1/6)
Wentao Guo retweeted
🎉 Thrilled to share that our paper, “Thought-Level Beam Search for Reasoning,” has been accepted to COLM 2026!
Different from Self-Consistency and Pruning-only approaches, Gambit actively reallocates test-time compute while generation is in flight — pruning low-scoring reasoning traces and branching from the strongest prefixes. See thread 🧵
Wentao Guo retweeted
Robot policies can move but can't think.
LLMs can think but can't move.
So we connected them.
Real robot: 16.7% → 97.3%
Sim (LIBERO-PRO): 12.8% → 53.3%
This was a large cross-domain effort, made possible by collaborators across systems, language models, robotics, AI for Science, trustworthy learning, and more.
Huge thanks to everyone who helped build, validate, and test MLS-Bench:
Bohan Lyu, Yucheng Yang, Siqiao Huang @KnightNemo_, Jiaru Zhang, Qixin Xu, Xinghan Li @XinghanLi66, Xinyang Han, Yicheng Zhang, Huaqing Zhang @zhqwqwq, Runhan Huang @RunhanH, Kaicheng Yang, Zitao Chen, Wentao Guo @WentaoGuo7, Junlin Yang @junlin45300, Xinyue Ai @KeelyAi04, Wenhao Chai @wenhaocha1, Yadi Cao @YadiCao, Ziran Yang @__zrrr__, Kun Wang @KunWang0129, Dapeng Jiang, Huan-ang Gao, Shange Tang @sangertang1999, Chengshuai Shi @chengshuai_shi, Simon S. Du @SimonShaoleiDu, Max Simchowitz @max_simchowitz, Jiantao Jiao @JiantaoJ, Dawn Song @dawnsongtweets, and Chi Jin @chijinML. (8/9)
Wentao Guo retweeted
Why do we store the SSM state at all?
More and more models are hybrids (Nemotron-3, Qwen3.5), so SSM decode speed matters.
We only write it back every step so the next step can read it.
ReplaySSM caches the recent inputs instead and rebuilds the state on the fly.
Same outputs, half the memory traffic
→ ~2x on spec decode at large batch sizes, which barely even helped SSMs before
→ up to 1.43x standard decode on large hybrids (up to Nemotron-Ultra-550B)
Work with @tri_dao
Blog + Code👇
Wentao Guo retweeted
Excited to share that LessIsMore has been accepted to ICML 2026! 🚀
LessIsMore is a training-free sparse attention for efficient long-horizon reasoning. By enforcing cross-head unified token selection, it brings up to 1.6x E2E speedup while preserving reasoning accuracy under practical workloads.
Huge thanks to my amazing co-authors and mentors @Jackfram2, @JiaZhihao, Ravi!
Paper: arxiv.org/abs/2508.07101
Code: github.com/DerrickYLJ/LessIs…
#ICML2026 #LLM #EfficientAI
Wentao Guo retweeted
We made Muon run up to 2x faster for free!
Introducing Gram Newton-Schulz: a mathematically equivalent but computationally faster Newton-Schulz algorithm for polar decomposition.
Gram Newton-Schulz rewrites Newton-Schulz such that instead of iterating on the expensive rectangular X matrix, we iterate on the small, square, symmetric XX^T Gram matrix to reduce FLOPs. This allows us to make more use of fast symmetric GEMM kernels on Hopper and Blackwell, halving the FLOPs of each of those GEMMs.
Gram Newton-Schulz is a drop-in replacement of Newton-Schulz for your Muon use case: we see validation perplexity preserved within 0.01, and share our (long!) journey stabilizing this algorithm and ensuring that training quality is preserved above all else.
This was a super fun project with @noahamsel, @berlinchen, and @tri_dao that spanned theory, numerical analysis, and ML systems! Blog and codebase linked below 🧵
Wentao Guo retweeted
Introducing M²RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
We bring back non-linear recurrence to language modeling and show it's been held back by small state sizes, not by non-linearity itself.
📄 Paper: arxiv.org/abs/2603.14360
💻 Code: github.com/open-lm-engine/lm…
🤗 Models: huggingface.co/collections/o…
Wentao Guo retweeted
The newest model in the Mamba series is finally here 🐍
Hybrid models have become increasingly popular, raising the importance of designing the next generation of linear models.
We've introduced several SSM-centric ideas to significantly increase Mamba-2's modeling capabilities without compromising on speed. The resulting Mamba-3 model has noticeable performance gains over the most popular previous linear models (such as Mamba-2 and Gated DeltaNet) at all sizes.
This is the first Mamba that was student led: all credit to @aakash_lahoti @kevinyli_ @_berlinchen @caitWW9, and of course @tri_dao!
Wentao Guo retweeted
FA4 now available in lm-engine: github.com/open-lm-engine/lm…
13.4% end-to-end speedup for Llama 8B training on 4x GB200s (1 node) 🚀🚀🚀
1005.55 TFLOPs for SDPA vs 1140.73 for FA4 (BF16 precision)
@tedzadouri @ultraproduct @__tensorcore__ @tri_dao cooked
Thanks to @bharatrunwal2 for running the experiment!
Wentao Guo retweeted
Asymmetric hardware scaling is here. Blackwell tensor cores are now so fast, exp2 and shared memory are the wall. FlashAttention-4 changes the algorithm & pipeline so that softmax & SMEM bandwidth no longer dictate speed. Attn reaches ~1600 TFLOPs, pretty much at matmul speed!
joint work w/ Markus Hoehnerbach, Jay Shah(@ultraproduct), Timmy Liu, Vijay Thakkar (@__tensorcore__ ), Tri Dao (@tri_dao)
1/
Wentao Guo retweeted
We identified an issue with the Mamba-2 🐍 initialization in HuggingFace and FlashLinearAttention repository (dt_bias being incorrectly initialized).
This bug is related to 2 main issues:
1. init being incorrect (torch.ones) if Mamba-2 layers are used in isolation without the Mamba2ForCausalLM model class (this has been already fixed: github.com/fla-org/flash-lin…).
2. Skipping initialization due to meta device init for DTensors with FSDP-2 (github.com/fla-org/flash-lin… will fix this issue upon merging).
The difference is substantial. Mamba-2 seems to be quite sensitive to the initialization.
Check out our experiments at the 7B MoE scale: wandb.ai/mayank31398/mamba-t…
Special thanks to @kevinyli_, @bharatrunwal2, @HanGuo97, @tri_dao and @_albertgu 🙏
Also thanks to @SonglinYang4 for quickly helping in merging the PR.
Wentao Guo retweeted
Gave a talk at @GoogleDeepMind on our joint work with @HazanPrinceton and @tri_dao on deployable RL policies for robotics. Great discussion with @JeffDean.























