Replaying a production trace against a new version of an agent works until the first step where the new version does something different. Usually that's step two or three. Here's the shape of it, and then the three things you actually need to compare two versions.
Say you have a coworker agent in Slack. Someone in finance writes "put together the revenue section for Thursday's board deck". The agent queries the warehouse, pulls the billing export, reconciles the two, builds the spreadsheet, writes the slides, posts a draft in the channel, takes comments, revises, and files the follow ups as tickets. Two days, thirty odd tool calls, a few rounds with the finance lead. It's been in prod for months.
Someone changes the prompt so the agent reuses queries it already ran this month instead of pulling everything again. Faster and cheaper, which was the whole point. The eval set is 50 graded tasks, scored on whether the numbers in the deck match the warehouse and whether the slides read well. All 50 pass, so it ships.
Two weeks later the CFO asks why the board saw a revenue number 6% above the billing export.
The new version skipped the billing reconciliation whenever the warehouse looked consistent with last month, and a refund batch hadn't landed in the warehouse yet. The slides were clean and the finance lead's comments were about the churn chart. The eval scored the deck. Nobody scored the path.
So you try the obvious fix. Rerun last month's tasks against the new version with the recorded tool responses. It breaks at the reconciliation step, because the recording has the billing export from a call the new agent never makes. From there the state is wrong, every later response belongs to a run that didn't happen, and the finance lead's "looks good, add churn" was a reaction to a draft this version didn't produce. The part after the fork is the only part you care about, and it's the part you can't replay.
What you need instead
1. Sessions picked by what the change touches. A change to how the agent reuses work touches every step where it decides whether to check something again, so you pick the sessions where it checked before: reconciliations, lookups against a second system, any number that ends up in front of a person. A model swap touches more. A fixed golden set answers a different question.
2. A way to serve the calls the new version makes, and the review rounds it never triggers. Frozen responses where the step matches the recording. A sandbox, a simulated tool, or a simulated reviewer where it diverges. And the report has to say which one produced each result, because a simulated answer is a guess, not evidence.
3. A diff of actions, not scores. "Reconciliation skipped in 31 of 200 tasks" is a hold. "0.91 vs 0.92" tells you nothing about what the agent did.
We did this by hand for one team recently and found a failure they didn't know about. Now building it as an independent review that runs before a change ships, for long running agents, the ones that need thirty tool calls and a few rounds with a person to finish a job. Coworker agents, assistants, recruiting, support, ops.
Which action change would make you hold a release even if the output looked fine?
🤖 Made with AI
Prompt: Design an edgy, pixelated, retro-futuristic landing page for a 'Node-Based Gaming Asset Engine' waitlist. Predominantly white background, black text, and CRT Green (#39FF14) as the accent.
The hero uses an asymmetrical layout:
Left: Bold black pixel font headline NODE YOUR WAY TO SHIP. ('NODE' in CRT Green), sub-headline, black-bordered email input, and a CRT Green 'JOIN THE WAITLIST >' button with a pixelated urgency bar.
Right: Dynamic visual of a pixel-art node-based UI (black nodes, animated CRT Green flow lines) showcasing a rotating pixelated 3D gaming asset (grey/black with CRT Green highlights), all within a subtle pixel grid and blueprint-style overlays
Gemini 3.0 Pro in @FactoryAI
Current SOTA imho:
@FactoryAI with Gpt 5.1 for building
@cursor_ai and Sonnet 4.5 with built in browser for UI tweaks
