Aligning agent evals with your personal judgment is essential for building improvement loops you can trust. Otherwise, you’ll steer your coding agents towards problems that don’t exist, and miss the ones that do. A common approach for agent evals is LLM-as-a-Judge or Agent-as-a-Judge that labels your traces. The foundation of the eval is the prompt you define as a rubric. However, most people do not realize the first prompt they create doesn’t cover all cases that their judge will have to evaluate. Say you want to flag instances where users are frustrated with your agent. What happens if a user is still frustrated after an issue is resolved? If your initial prompt did not include clarification on whether to flag this or not, the judge might or might not label this trace as frustrated. When this ambiguity scales to thousands of traces, you'll be making changes to your agent based on false positives or negatives. Therefore, having a human in the loop to verify the evals is a natural prereq to creating scalable evals you can trust. This has traditionally looked like the following: hand-label agent traces, compare your labels with the judge’s, and continue iterating on the rubric until the judge aligns with your ground-truth labels. I work directly with agent engineers who have historically found this tedious and avoided it entirely, but I’d argue that aligning against human-reviewed traces is well worth the time. Labeling even just 10-20 traces will make the quality of your rubric significantly better, allowing you to reliably grade tens of thousands of traces. I’ve personally seen teams that neglected judge alignment and ended up spending entire sprints’ worth of engineering time making choices based on a false understanding, only to have to do a complete 180 after realizing their evals weren’t giving them accurate insights. This is why at @JudgmentLabs, we spent a lot of time streamlining this process in our product. Once you provide the ground-truth labels, our agents iteratively refine and backtest the rubric until it matches your labels. The product also reminds you to regularly label new traces so that your judges stay accurate as your agent evolves. Whether you’re using Judgment Labs or not, aligning your judges’ rubrics will not only improve the quality of their evals, but will eventually help create an eval system you can trust and scale.

we need more fantasy football heads

When @henryxiao7 sent me this in June, I didn’t know how serious he was about either offer. 3.5 months into my time at Judgment Labs, he was not joking around. I’ve worked on 30+ deployments, helping teams build evals they can actually trust for their agents. I’m also 7-0 in our company fantasy league. Today Judgment is generally available. If you’re building agents, try it out. I’d love to help you get your evals right too.