Keep an agent evaluation set current as the real world changes
Review evolving reference cases and test each agent update against current, verifiable expected behavior.
How this works
Review evolving reference cases and test each agent update against current, verifiable expected behavior. Use this when an agent's real-world reference cases change and a previously correct answer may now be stale. The benchmark should test current expected behavior while preserving history so updates do not hide regressions.
Maintain versioned cases with evidence, dates and specialist-reviewed expectations. Separate changed facts from changed evaluation criteria, then test the proposed agent version against both stable and refreshed cases. Passing the set establishes only its documented coverage.
Before you start
Existing benchmark
Obtain case IDs, expected behavior and prior run results.
Reference evidence
Identify authoritative sources and review dates for changing facts.
Evaluation environment
Use fixed agent configuration and available test tools.
Prepare the inputs
- cases
- Required
ID, input, expected behavior and evidence
Record freshness and coverage category.
- change_candidates
- Required
new facts, statuses or discovered failures
Verify changes before updating expected outputs.
- agent_versions
- Required
baseline and proposed configuration
Preserve model, prompt and tool versions where available.
- review_rules
- Required
specialist approval and scoring criteria
Separate factual correctness from formatting preferences.
Implementation, step by step
Audit freshness
- Find cases whose evidence or status has changed.
- Keep stable cases unchanged to preserve comparison value.
Checkpoint: A refresh queue identifies justified updates.
Verify expectations
- Review new evidence and revise only affected expected behavior.
- Record old and new expectations with the reason and reviewer.
Checkpoint: The benchmark has a traceable version.
Run comparisons
- Execute baseline and candidate configurations on the same case version.
- Capture actual outputs, tool failures and run conditions.
Checkpoint: Comparable results are available.
Inspect failures
- Classify factual, retrieval, tool, instruction and format failures.
- Check borderline scores manually and avoid letting the tested agent be the sole judge.
Checkpoint: The regression report explains meaningful differences.
Prepare release judgment
- Summarize coverage, unresolved failures and any accepted limitations.
- Retain the benchmark and run artifacts for later comparisons.
Checkpoint: The owner can make an informed release decision.
Decisions along the way
Reference fact changed
Update the case with evidence instead of calling the old answer a model regression.
Scoring rule changed
Version the rubric and avoid comparing scores as if unchanged.
Candidate passes but coverage is narrow
State the limit and add relevant cases before broad conclusions.
What you will produce
benchmark-version.json
JSONA maintained reference set.
Include: case_id · input · expectation · evidence · review_date · version
agent-regression-report.md
MarkdownA reviewable comparison for release decisions.
Include: versions · results · failures · coverage · limitations
Check the result
- Verify changed expectations against current evidence.
- Confirm both agent versions use the same benchmark revision.
- Inspect failed and sampled passing cases manually.
- Check no test answer leaked into the agent's task context unintentionally.
When something goes wrong
Reference source unavailable
Mark the case stale or unverified rather than invent an update.
Tool outage affects runs
Separate infrastructure failures and rerun under comparable conditions.
Judge disagrees with specialist
Retain both assessments and use the reviewed rubric.
A worked example
SYNTHETIC EXAMPLE · NOT A REPORTED BUSINESS RESULT
Request
Synthetic: refresh an eligibility research case.
Inputs
Case C4 previously expected status open. The authoritative source now says closed as of 1 July. Candidate agent still reports open.
Expected result
Synthetic result: version C4 with the dated source, rerun both agents and report the stale-status failure against the refreshed expectation. Preserve the prior case history.
Run a small pilot
Start small
Refresh a small set with stable and changing reference cases.
Record the baseline
Record review effort and regressions missed by the old set.
Agree success criteria
Agree coverage and failure-resolution criteria before release reliance.
Stop or revise
Stop score comparisons when benchmark or configuration versions cannot be reproduced.
Keep the scope clear
- A passing benchmark does not prove absence of hallucinations or failures.
- Benchmark maintenance does not authorize production release.
Check before using
Benchmark performance is limited to its coverage. Do not equate a passing set with absence of hallucinations.
Business activities
- Operate and improve AI workflows
FROM A RECIPE TO YOUR WORKFLOW
Bring the company context together.
This workflow needs reliable inputs before it needs a better prompt. Moat8 Vault brings working information into a readable archive for the AI tools you use.
For this recipe: Representative evaluation cases · Current source-of-truth evidence · Proposed agent update
Start with one recurring task, agree which sources to use, and review the first result with your team.
Discuss this workflow