tushar

Picking a good inference provider is difficult. Picking the best inference provider for an open model is surprisingly hard: The reason: no single provider does it all. On OpenRouter, GLM 5.3’s GPQA score ranges from 78% to 89% (quite a large gap, despite the caveats†), and the trends in GPQA is not strongly correlated with other tasks, so it varies by workload. Why discuss this now? Recently, Jon Barron and colleagues showed that standard softmax implementations hide two silent bugs: Softmax Collapse, where the target gradient rounds to zero, and Softmax Cancellation, where identical logits still produce a gradient. Widely speaking, inference has the same problem, except it's harder to get something agreed to be a 'correct' outcome. In my experience, providers on alternative chips tend to have worse silent framework bugs, though these issues are apparent on NVIDIA inference frameworks as well. MoE models are especially fragile: the router bias and router output need fp32, etc. (there's a long list of half-patched issues). As deployment engines grow more complex, these bugs get almost exponentially harder to catch. (Note†: SGLang or vLLM in their default configurations should give the best accuracy. The issue come from what providers add, like some custom quantization, speculative decoding, and custom kernels, or even simply forgetting to update to the latest versions of the frameworks that improve accuracy. And consider the case where you have no open reference at all (i.e they take care of both post-training and inference, so you don't own your own canon copy of the model, where you wouldn't know if there's some error in the stack silently reducing your quality).