Check research-agent changes against a fixed evaluation checklist
A practitioner compared research workflows using a known example and explicit completion criteria before keeping changes to prompts or connected tools.
How this works
A practitioner compared research workflows using a known example and explicit completion criteria before keeping changes to prompts or connected tools. Use this when changing a research agent's prompt or connected tools and the team needs to know whether the workflow still completes its actual task. The check should inspect evidence and saved artifacts, not only a polished final answer.
Define a fixed research request and explicit criteria for source quality, synthesis, provenance, saved outputs and completion. Run baseline and candidate setups, inspect their artifacts independently and compare failures. A small example is a regression screen rather than a comprehensive factual benchmark.
Before you start
Reference task
Choose a representative request with verifiable requirements.
Completion checklist
Define source, synthesis and artifact expectations.
Comparable setups
Record baseline and candidate prompts, tools and environment.
Prepare the inputs
- research_request
- Required
fixed task and scope
Preserve requested sources, dates and deliverables.
- evaluation_checklist
- Required
observable criteria and hard failures
Avoid grading only style or length.
- configurations
- Required
prompt, tool and model identifiers
Record differences explicitly.
- reference_evidence
- Required
known facts or acceptable source classes
Do not reveal expected answers to the tested workflow unnecessarily.
Implementation, step by step
Define observable completion
- Translate the request into checks for research coverage, claims and output files.
- Include negative cases such as unavailable evidence.
Checkpoint: The checklist is usable by an independent reviewer.
Run each setup
- Execute the same task under comparable conditions.
- Capture tool errors, source access and saved artifacts.
Checkpoint: The comparison has complete run evidence.
Inspect source grounding
- Open cited sources and verify important claims.
- Distinguish inaccessible sources from sources actually read.
Checkpoint: Factual checks have primary evidence.
Inspect workflow outputs
- Verify requested files exist, open correctly and contain the promised material.
- Check that the final response accurately describes what was completed.
Checkpoint: Completion claims are reconciled.
Compare and recommend
- Identify regressions and useful changes with concrete examples.
- Keep unresolved gaps visible and rerun after targeted corrections.
Checkpoint: The owner can decide whether to keep the change.
Decisions along the way
A source becomes unavailable between runs
Record the environmental difference and avoid attributing it solely to the agent.
Output exists but is incomplete
Mark the artifact requirement failed.
Agent grades its own answer as correct
Use independent source and artifact inspection.
What you will produce
research-regression-report.md
MarkdownA concrete comparison of workflow completion.
Include: task · configurations · criteria · results · failures · limits
research-run-manifest.json
JSONA reproducible record of each run.
Include: run_id · sources · artifacts · errors · configuration
Check the result
- Open important sources and verify claims directly.
- Inspect actual files rather than trust reported paths.
- Confirm baseline and candidate received the same task.
- Check coverage limits and environmental differences are stated.
When something goes wrong
Tool access differs unexpectedly
Align environments or report the comparison as limited.
Artifact cannot be opened
Record the failure and require a valid output before passing.
Expected facts are outdated
Refresh the reference evidence without hiding prior results.
A worked example
SYNTHETIC EXAMPLE · NOT A REPORTED BUSINESS RESULT
Request
Synthetic: test a supplier-research workflow.
Inputs
Task requires three primary sources and a saved comparison table. Candidate gives a fluent answer with three links but writes no table file.
Expected result
Synthetic result: source checks may pass individually, but artifact completion fails. Do not accept the run because the final message says the table is ready.
Run a small pilot
Start small
Compare one representative task and one missing-source variation.
Record the baseline
Record manual review effort and failures hidden by final-answer polish.
Agree success criteria
Agree completion criteria before using the screen for changes.
Stop or revise
Stop release conclusions when the checklist or environment is not comparable.
Keep the scope clear
- A regression example does not establish general factual accuracy.
- Testing a setup does not authorize production configuration changes.
Check before using
Review the evidence yourself; an agent grading its own output can miss shared errors. The example does not establish a comprehensive factual-accuracy benchmark.
Business activities
- Research customers and markets
- Operate and improve AI workflows
FROM A RECIPE TO YOUR WORKFLOW
Bring the company context together.
This workflow needs reliable inputs before it needs a better prompt. Moat8 Vault brings working information into a readable archive for the AI tools you use.
For this recipe: Representative research task · Expected evidence and artifact requirements · Candidate prompt or tool changes
Start with one recurring task, agree which sources to use, and review the first result with your team.
Discuss this workflow