Why temperature 0 isn't deterministic in LLM inference? (a popular technical LLM interview question) At temperature 0, inference engines usually switch to greedy decoding directly. At every step, they pick the highest-logit token, so the same prompt should give the same output every time. In practice, it doesn't. Thinking Machines actually tested this by sending "Tell me about Richard Feynman" to Qwen3-235B 1,000 times at temperature 0 and got 80 unique completions. While picking the biggest logit is a perfectly deterministic step... ...the variation actually comes from computing the logits themselves. If those numbers come out slightly different, argmax can pick a different token. Let's understand how the logits can slightly change at times. 1) The order of addition changes the result: GPUs store numbers with limited precision. fp16 keeps only 11 significant bits, and between 1024 and 2048, that leaves no room for anything after the decimal point. So 1024.4 gets rounded back to 1024. Now consider 1024 + 0.4 + 0.4 in two orders. Left to right, 1024 + 0.4 rounds back to 1024, and adding 0.4 again rounds back to 1024. The answer is 1024. If the small numbers go first, 0.4 + 0.4 gives 0.8, and 1024 + 0.8 rounds up to 1025. The answer is 1025. An LLM performs billions of such additions in every forward pass, so if their order changes, the logits change slightly too. 2) What causes the order to change? The popular reason is that parallel GPU threads finish in an unpredictable order. But that isn't the main reason, since the kernels in a typical forward pass compute each sum in a fixed order. So the same matrix multiplication on the same inputs, at the same batch size, gives bit-for-bit identical results every time. 3) The real reason is the other requests on the server: An inference server groups requests from many users into a batch and runs them together. The kernels decide how to split their work based on the batch size. - With a large batch, every GPU core is busy, so each core computes its sum in one go. - With a small batch, many cores would sit idle, so the kernel cuts each long sum into chunks, spreads them across cores, and combines the partial results at the end. A different split leads to a different order of addition, and the batch size depends on how many other people are using the server at that moment. 4) How this tiny difference changes the entire answer: Consider two runs of the same prompt: In a batch of 8, say the logits come out as: - "Queens" = 12.3417 - "New" = 12.3415 In this case, argmax picks "Queens". In a batch of 32, the rounding works out slightly differently, giving: - "Queens" = 12.3414 - "New" = 12.3416 In this case, argmax picks "New". The output only flips at such near ties. And every token after that is generated from a different prefix. In the above test by Thinking Machines, all 1,000 runs matched for the first 102 tokens. At token 103, 992 runs wrote "Queens, New York" while 8 wrote "New York City". 5) The solution: This isn't a big problem per se, but to make it genuinely deterministic, every kernel should split its work the same way regardless of batch size (for attention, that means a fixed chunk size), even if it leaves some cores idle. Thinking Machine tested this with batch-invariant kernels, where all 1,000 runs were identical. But this impacted speed. On Qwen3-8B, default vLLM took 26s for 1,000 requests, while the deterministic version took 55s, and 42s after an improved attention kernel. The visual below depicts what we discussed. That said, batching isn't the only thing that changes how these sums get split. A cache hit does too, since some attention kernels process cached tokens separately from new ones, and Thinking Machines had to handle this case explicitly. I recently covered the four caches in LLM serving in an article, with code for each. Read it below.