Good judges are the ones that make it easy for other agents to answer the questions you have about how your agent performs. Let’s take a prompt that you might give to an agent to analyze a set of agent traces: “Looking across the past month, where were users frustrated with my agent across different user and request types, what kinds of problems were causing their frustration, and were their problems resolved?” An agent analyzing these traces will struggle to answer this question. The scale of the question involves searching across millions of traces and digging through various semantic details across all of the traces on that timescale. SOTA coding agents still struggle with running analysis at this scale when traces aren’t annotated. Getting your agent to do more accurate, deep, and token efficient analysis is conceptually the same idea as adding indexes to a database. Without indexes, queries have to do a full table scan, which is inefficient. Evaluating and analyzing agent traces is the exact same idea! The labels produced by judges become the indexes that analyzer agents can use to search across trace data more cheaply and efficiently. Imagine looking through a million agent runs by hand. Now imagine you had a table of contents you could flip through instead! Going back to the example question, the way that I’d approach having this question answered is to create a judge that indexes my data on each dimension I ask about in the question: “Where were my users frustrated” → User Sentiment Judge “across different user” → User Type “and request types” → Request Type “what kinds of problems were causing their frustration” → Failure mode 1, Failure mode 2, etc. “and were their problems resolved?” → Resolution Status With these judges labeling my traces, instead of an agent having to read every trace in full and reason over each of them, it’s searching via indexes across spans, traces, sessions and with agent/LLM judges, a citation explaining why a trace received its label. Like a database index, the labels are pre-computed once instead of every time you ask a question, which is what makes the analysis both cheaper and more accurate. The agent we offer at @JudgmentLabs filters and groups your traces with these labels and only views cited traces when it needs specific details. Its analysis goes straight to your coding agent as context through Slack and/or MCP to ship fixes and improvements to your agent. Regardless of where you run your evals, it’s important that you’re setting them up to answer the question that you actually care about.