Hongyang Zhang

This technique makes LLM inference 2-3x faster: (Google Search uses this in production) Under autoregressive decoding, the target model produces one token per forward pass. Each token is conditioned on the sequence generated so far, so generation cannot advance until another pass runs through every transformer layer. At low batch sizes, this process is usually memory-bandwidth bound, leaving much of the GPU's compute idle. Instead, a cheaper draft path can propose several future tokens. The large target model can then check the complete block in one forward pass (this is similar to prefill). This technique is called speculative decoding. Google uses it in production for AI Overviews in Search. Suppose the drafter proposes five tokens. If the target accepts all five, it can also add one bonus token from the same verification pass. If the fourth token is wrong, the first three are accepted. The target corrects the fourth, and the remaining draft is discarded. With the proper acceptance rule, the output distribution remains identical to the target model's distribution. There are four common ways to generate drafter tokens: 1. Two-model speculative decoding A smaller model generates a few tokens sequentially, and the larger model verifies them together. This is the most basic setup since it leaves the target unchanged. The tradeoff is that you must maintain another set of weights, another KV cache, and more scheduling work. 2. EAGLE EAGLE replaces the separate draft LLM with a lightweight module trained on the target model's hidden states. It drafts from the target's internal representation, but the module must be trained for that checkpoint. 3. Medusa Medusa attaches several prediction heads to the target model. Each head predicts a different future position, so several candidate tokens are produced in parallel. The heads do not condition on each other, so their predictions may not form one consistent sequence. Medusa builds a tree and verifies several candidate paths together. 4. LayerSkip LayerSkip uses the target model's early layers as the drafter. The remaining layers verify and correct the proposed tokens. This removes the second model and additional prediction heads. However, it requires a checkpoint trained for reliable early exits using layer dropout and an early-exit loss. In production, the useful metric is accepted tokens per target pass after accounting for drafting time, verification overhead, and extra memory. A more accurate drafter can still make the complete system slower if producing its guesses costs too much. To dive deeper, I wrote a deeper breakdown of speculative decoding, including the acceptance process and why the output distribution remains unchanged. Read it below.