@_rishabhranjan_

Stanford CS PhD w @jure and @guestrin. Prev. CMU w @zacharylipton, IIT Delhi. Deep learning fanatic.

 Stanford, CA
rishabh-ranjan.github.io
Joined March 2022
  • Tweets91
  • Following243
  • Followers322
10Media
Our new work, TokenGate, brings MoE-like conditional computation to context tokens, resulting in up to ⚡️11x⚡️ faster prefill and ⚡️3x⚡️ lower memory!
Liana
@lianapatel_Oct 21
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
2
2
9
631
rishabh ranjan retweeted
Liana
@lianapatel_Oct 22
Jev + TokenGate would be such a game changer 🔥🔥 Insanely fast decode with Jev + insanely fast prefill with TokenGate… someone should really try it 👀
rishabh ranjan
@_rishabhranjan_Oct 22
Jev and "decision models" have shown the importance of LLM prefill optimization. Now 🧨TokenGate🧨 unlocks a whole new frontier of efficient prefill by e2e-learning to drop tokens on-the-fly 🤯 Jev+TokenGate=💥💥💥? @typesafeai @CompleteSkeptic @EGafni @hackgoofer @karpathy @willdepue @swyx
4
18
1,065
Jev and "decision models" have shown the importance of LLM prefill optimization. Now 🧨TokenGate🧨 unlocks a whole new frontier of efficient prefill by e2e-learning to drop tokens on-the-fly 🤯 Jev+TokenGate=💥💥💥? @typesafeai @CompleteSkeptic @EGafni @hackgoofer @karpathy @willdepue @swyx
Liana
@lianapatel_Oct 21
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
3
2
14
2,221
Token visualizations like these intuitively show that Token-Gated LLMs indeed learn to skim 🤓 Crazy how TokenGate induces smart token selection behaviors adapted to the task: QA needs local info, summarization needs global info ... the same TokenGate can do both 🤯🙇
Liana
@lianapatel_Oct 21
Replying to @lianapatel_
(2/n) Expert human readers skim large inputs and closely read only what matters. But LLMs can't: they process every input token through every layer, whether or not it carries information. For long-context inputs (e.g., code, long documents, research articles), prefill is inherently costly and wasteful. TokenGate now enables e2e learnable selection over context: LLMs learn which tokens to process by skimming.
1
6
281
end-to-end learning ftw 🙌 as always 😌
Liana
@lianapatel_Oct 21
Replying to @lianapatel_
(7/n) Why do TokenGated LLMs win? 🏆🏅 TokenGate flexibly learns different token-selection patterns in a task-dependent manner. eg) it keeps answer keywords for QA and stop words at the end of phrases for summarization. And the backbone co-adapts with the gate, becoming robust to pruning and re-distributing information among tokens. Both effects are possible because TokenGate enables e2e learning.
4
226
rishabh ranjan retweeted
rishabh ranjan
@_rishabhranjan_Oct 21
Replying to @kartikb753 @lianapatel_
TokenGate is extremely good at needle style tasks. Please see our NIAH (needle-in-a-haystack) results! The secret is that no token is lost completely, the inital layers can merge its info into the kept tokens!
1
2
103
Very interesting work on pruning input tokens! It’s similar to how we do latent tokenization in hierarchical models, except here the unnecessary tokens are dropped instead of being mean-pooled.
Liana
@lianapatel_Oct 21
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
1
3
16
1,046
rishabh ranjan retweeted
Vignesh Kothapalli
@kvignesh1420Oct 21
The bitter lesson of scaling applies when systems are learnable end-end. Kudos to @lianapatel_ and @_rishabhranjan_ for bringing it down to the token selection stage of a forward pass. Paper: arxiv.org/abs/2610.26754

TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill...

Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters. On the other hand, language models process all input tokens...

arxiv.org
Liana
@lianapatel_Oct 21
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
1
4
9
942
rishabh ranjan retweeted
Stanford AI Lab
@StanfordAILabOct 21
Token-Gated LLMs achieve 11x faster processing over long-context inputs by learning to skim context and only forward tokens that matter. We saw this paradigm shift before when going from dense models to MoEs, and now TokenGate enables another generation of efficient LLMs!
Liana
@lianapatel_Oct 21
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
9
19
132
10,425
rishabh ranjan retweeted
Connor Shorten
@CShorten30Oct 21
Replying to @lianapatel_
Game changing! Congratulations Liana and team! 🔥
1
1
4
387
TokenGate follows on the footsteps of the Mixture-of-Experts layer: arxiv.org/abs/1701.06538 Hat-tip to @NoamShazeer @Azaliamirh @MaziarzKris @quocleix @geoffreyhinton @JeffDean for showing how to scale up model capacity with conditional computation and revolutionizing the field 🙇

Outrageously Large Neural Networks: The Sparsely-Gated...

The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been...

arxiv.org
1
2
7
363
Also indebted to works like Switch transformer (@LiamFedus @barret_zoph), Mixtral (@AlbertQJiang @arthurmensch @dchaplot @sophiamyang @Fluke_Ellington @theo_gervet) and DeepSeekMoE (@deepseek_ai) which kept the flame alive ❤️‍🔥
1
4
163
rishabh ranjan retweeted
Shina Patel, MD@shina1812Oct 21
Replying to @lianapatel_
Conditional computing over context tokens is fire 🔥
1
1
2
280
rishabh ranjan retweeted
rishabh ranjan
@_rishabhranjan_Oct 21
Our new work, TokenGate, brings MoE-like conditional computation to context tokens, resulting in up to ⚡️11x⚡️ faster prefill and ⚡️3x⚡️ lower memory!
Liana
@lianapatel_Oct 21
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
2
2
9
631
rishabh ranjan retweeted
rishabh ranjan
@_rishabhranjan_Oct 21
TokenGate follows on the footsteps of the Mixture-of-Experts layer: arxiv.org/abs/1701.06538 Hat-tip to @NoamShazeer @Azaliamirh @MaziarzKris @quocleix @geoffreyhinton @JeffDean for showing how to scale up model capacity with conditional computation and revolutionizing the field 🙇

Outrageously Large Neural Networks: The Sparsely-Gated...

The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been...

arxiv.org
1
2
7
363
rishabh ranjan retweeted
Liana
@lianapatel_Oct 21
very cool to see TokenGate already trending on @askalphaxiv — and only a few hours since our release 👀 🚀super excited for this work. check it out and share your questions! (links in comments)
Liana
@lianapatel_Oct 21
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
1
9
32
3,133
rishabh ranjan retweeted
Liana
@lianapatel_Oct 21
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
29
64
13
361
43,521
rishabh ranjan retweeted
Daman Arora
@amuseddamanOct 16
1/ RL for improving task success has made LLMs remarkably capable. But what happens when the model fails? Turns out, training for graceful failure can be surprisingly tricky! Very excited to share our new paper: Training Language Models to Fail Gracefully 🧵
23
89
19
479
57,889
rishabh ranjan retweeted
Jure Leskovec
@jureSep 29
I’m very excited to share @nvidia Kumo Tabular, a new family of foundation models for tabular data. Kumo Tabular establishes the new Pareto frontier across the entire accuracy–inference-time tradeoff. Just as importantly, we are releasing it openly: open weights, open-source software, and a permissive license for commercial use. HuggingFace: huggingface.co/nvidia/Kumo-T… GitHub: github.com/NVIDIA/structured…
32
155
34
1,106
128,438
rishabh ranjan retweeted
Liana
@lianapatel_Aug 31
Looking forward to VLDB in Boston this week! I’ll be on the job market this year for CS faculty positions. I’m also co-organizing the NOVAS workshop tomorrow, featuring research at the intersection of data systems and AI, including semantic operator optimization and agentic data systems. Please reach out if you’re attending—I’d love to meet and chat!
1
5
45
5,680
Load more