These number are such BS. There’s no justification for bumping down the numbers if OSS models while keeping American closed models essentially the same. I’ve really lost faith in these folks .

Pro tip for anyone reading quant KLD numbers (all credit to @YourLocalAILab for pushing people to report KLD in the first place). Before you trust a KLD number, look for four things: how many windows were used, how many tokens were in each window, how many total tokens were scored, what dataset is was for the fixed text, and how much context each scored token had behind it. If those aren’t listed, be skeptical and consider the source. Why window length matters A quant’s rounding error on any single step is small. But every new token looks back over the whole context (subject to how attention architecture works, ofc, this is just generally speaking), and in a quant, the stored information for every one of those earlier tokens carries a little error. The longer the context, the more noisy history the model leans on, and the more chances it has to drift from the base model. The research backs this up. Mekala et al. (EMNLP 2025) tested quants on inputs over 64K tokens. 8-bit held up, with under a 1% drop on average, while 4-bit methods lost as much as 59% on long-input tasks, and retrieval got worse the longer the input got. So a 256-token window tells you how a quant handles a quick chatbot exchange (very quick less than 200 words). It won’t give you a good picture of divergence from the base model at the lengths where people reason, research, code, or run long agentic jobs. It’s like looking at the clouds over your house in Kentucky and calling it the forecast for the whole South. You might be right about Kentucky, but you have no idea it’s sunny in Florida. Check to see how tokens are scored Some people use 2,048-token windows but only score small chunks of them. What matters is how much context sat behind each scored token. Scoring the back half of a long window is fine; llama.cpp’s tool does this on purpose so every scored token has plenty of history (see github.com/ggml-org/llama.cp…) . Chopping text into 256-token pieces, or scoring only the first few hundred tokens of a window, throws out the part of the test where a quant is most likely to slip. That’s like taking a 100-question test, grading only the first 15 that you got right, and reporting an A plus as your final score. The bootstrap rope-a-dope You’ll see things like “Large X block-bootstrap resamples.” Bootstrapping is a real tool, and it tells you how noisy your number is. The large X, say 10,000, is how many times the same data got reshuffled, and the amount of actual text stays exactly the same. If the windows are too short, bootstrapping just hands you a very confident estimate of short-context divergence. It doesn't route out the error in the bias of that number. Averages can hide the damage Dettmers et al. (LLM.int8()) showed that big models lean hard on a tiny set of outlier features. Zeroing out roughly 0.1% of features made perplexity 600 to 1000% worse. Small errors in the wrong place matter a lot. Quant error is also uneven: most tokens look fine and a few go badly wrong, and an average KLD can bury those few. Ask for the tail numbers too, like 99th and 99.9th percentile KLD and how often the quant picks the same top token as the base. Different quant schemes approach the outliers differently (like hadamard rotations for instance). Here's the TL Dr of this: At least 2,048 tokens of context per scored token. Ideally, test at the lengths you actually use, but you can be limited by hardware here (we can't all rent b300s lol). Report the number of windows, tokens per window, total tokens scored, and which positions were scored. Use a big, varied dataset. One kind of text only measures one kind of text. Report the tails, and KLD by position if you can, alongside the average. These are where things are moving. Pair KLD with real long-context tests (like needle in a haystack tests, or the LAVD, estonia and hotel tests from @YourLocalAILab for instance, there are, ofc, other valid tests as well). KLD is measured on fixed text, so it can’t see small mistakes snowballing when the model is generating on its own over a long task. Compare against the model as released. That’s usually BF16, though some recent releases like Kimi ship in native 4-bit. Local folks may not be able to score millions of tokens, and that’s fine. Just say what you did. The biggest key is transparency and alot of people don't want to show that, and hope a shiny number will get folks in, and that's just misleading and not where the literature is going.

Tuning a glm 5.3 flash quant for flare. :-)

So proud of our team. We’ve been cooking for a while. This is what OSS work looks like. Pushing the frontier on models you can run locally. Improving the local ai experience every single day.

Those later two brought to you by @YourLocalAILab . Such cool work!

Such a terrible workflow. I can do all this on premises on my. Network. Not send any client data to anyone but myself. That’s KLC in a nutshell .

We have been hard work in the kitchen. Proud of the crew @YourLocalAILab .