Zima

Attached is the before and after results for testing Qwen3.8-Flash-Next-oQ4e-mtp on XHigh after significant changes to the bench. I made changes in the bench yesterday to mitigate the issue of the model inferring it was being graded. It worked and increased the model's ability to find defects - at least for Qwen3.8 Flash next. I addressed any naming that inferred testing, moved visible tests into hidden tests outside the sandbox (these were included originally to assist the model), removed references to Tiers. Basically cleaned/removed anything visible to the agent that might infer a test was occurring. It still occasionally infers testing/grading from recognizing that the repo looks synthetic, because it is. The 8 visible tests previously used but not scored in my output are now hidden. This means the testing scores moved up to 24 total tests versus 16. The model no longer has any ability to see those 8 tests it used to see. Also tightened the prompt a little. References to grader/hidden tests dropped dramatically. I had 27 references across 7 runs versus 30-40 per run prior. This does mean that the baseline starts over because the scoring has changed and it should. The results are obvious in the before scores from Sep 9th versus today. A jump of 20% in scoring. 68.8% - 84%. This is with a reasoning budget of 9k tokens. The reasoning exceptions per run were in the 2-3 range with the majority being 2. Snapshot & Compaction is a weakness for this model. I intend to run the 6-bit series again to see if the 84% number changes. What sent me down this path was that I had started testing mlx-serve and one of David's @ddalcu smaller quants and I needed a baseline in my mind to be able to compare to. This is what forced me to revisit this entire process. Happy I did because it was worthwhile 😊 Having the M5 has dramatically increased my ability to iterate quickly.