Ai2

Uh oh. If your AI benchmark uses an LLM-as-judge, you might want to read this. AllenAI tested six frontier models to see whether research ideas were actually novel. And tiny changes to the judge prompt caused crazy swings. GPT-5.4, same basic idea pairs “Which was judged novel?” → 93.5% “Which is novel?” → 40.9% “Which was judged more novel?” → 59.7% Across the study, changing the prompt moved accuracy by 👉50+ points 👈 in some cases and occasionally pushed judges below chance. So naturally they tried giving the judge more intelligence and tools. Didn't fix it. A GPT-5.6 Sol agent running xhigh reasoning with iterative literature search and no search cap scored → 0.78 macro-F1 Claude Opus 4.6 with only 5 pre-retrieved abstracts did about → 0.89 And the big agentic setup cost ~10× more per judgment. Two purpose-built novelty-evaluation systems lost to the researchers' cheapest simple prompted judge by → 8–29 macro-F1 points while costing up to → 32× MORE 🎯 Lesson Learned 👉 More model, more reasoning more RAG, and more money, doesn't automatically mean a better judge. AllenAI open-sourced the benchmark + code too, so now I want to see Qwen, DeepSeek and other local models run as the judges. The safer benchmark recipe is 👨‍🔬 build a human-rated calibration set 🤖 test the LLM judge against it 🔄 test multiple prompt/order variations 🗳️ aggregate multiple judgments 👀 send close calls back to humans Link in ALT