Replay recorded AI responses in integration tests
Create controlled response fixtures so application tests run reproducibly without making an unplanned live model call.
How this works
Your application calls an AI service in many integration tests. Slow and variable live responses make failures difficult to reproduce. Record approved request-response fixtures and replay them to test parsing, routing and recovery in the surrounding application.
This workflow verifies application behavior against known interactions. A replay pass does not establish that the live model still behaves the same way. Keep a separate, bounded live evaluation suite and make fixture refresh an explicit maintenance decision.
Before you start
Accountable owner
Engineering reviews the client-boundary change and the fixture update before merging.
Bounded pilot
Choose one team and a small historical sample before connecting a live recurring trigger. Confirm permitted records, the expected artifact, and a named reviewer.
Available access
Confirm access to the source systems listed below. If a connector is unavailable, use dated exports in a local folder and state which freshness checks remain manual.
Prepare the inputs
- client_call_inventory
- Required
Repository references and supported call shapes
Identify every client constructor, sync or async path and relevant provider option before editing.
- fixture_policy
- Required
Recording, replay and refresh rules
Define fields that affect request meaning, permitted normalization and sensitive-data handling.
- example_interactions
- Required
Approved synthetic or redacted requests and responses
Include valid outputs, malformed outputs, errors and streaming variants actually used by the application.
- test_scope
- Required
Named integration suite and network rules
Require strict replay in CI; unknown requests must fail rather than silently reach the provider.
Implementation, step by step
Map the client boundary
- Trace how the application initializes and invokes its AI client, including concurrency and streaming.
- Choose a narrow interception layer and preserve existing client behavior outside test mode.
Checkpoint: The team can name which calls are covered and which remain outside the fixture system.
Define request identity
- Build a canonical key from provider, model, messages, tools and settings that materially affect the response.
- Normalize only documented nondeterministic metadata; never drop a user field merely to make a cache hit.
Checkpoint: Equivalent requests match and meaningfully different requests remain distinct.
Record bounded fixtures
- Capture only approved interactions in an explicit recording mode and redact credentials before writing files.
- Represent repeated identical requests with different outcomes intentionally, including failure examples.
Checkpoint: Each fixture documents its origin, schema version and intended outcome.
Implement strict replay
- Return recorded responses without network access and fail clearly on missing or incompatible fixtures.
- Make concurrent reads safe and preserve the response shape expected by the original client.
Checkpoint: A test cannot incur a hidden live API call after a cache miss.
Exercise meaningful behavior
- Test success, malformed data, retry handling and provider-error paths through the real application boundary.
- Verify that normalization does not collapse different prompts or tool arguments to the same key.
Checkpoint: The suite catches application regressions rather than merely asserting fixture contents.
Maintain fixtures deliberately
- Attach a manifest of the request schema and application context used when recording.
- When context construction changes, review affected fixtures and refresh them through the explicit mode; keep old results available for comparison.
Checkpoint: Fixture freshness and live-model evaluation remain separate, documented concerns.
Decisions along the way
A replay fixture is missing
Fail the test with the canonical request summary; do not switch to live mode.
The same request can return multiple results
Store named or ordered variants for the tested paths instead of pretending the provider is deterministic.
What you will produce
fixture-manifest.json
JSONA versioned inventory of permitted recorded interactions.
Include: fixture_id · request_key · schema_version · recorded_at · redaction_status · expected_path
replay-validation.md
MarkdownA test report separating application replay coverage from live model quality.
Include: covered_calls · network_check · test_results · known_gaps · refresh_rule
Check the result
- Run the strict suite with provider network access disabled.
- Verify credentials and sensitive example content are absent from committed fixtures.
- Exercise at least one collision test for similar but meaningfully different requests.
- Check async and streaming behavior only where the application actually uses it.
When something goes wrong
Client signatures differ across call sites
Normalize at a supported adapter boundary and run existing tests before expanding coverage.
A fixture becomes stale after context changes
Fail with a clear version mismatch and refresh through the approved recording path.
Concurrency changes replay order
Use isolated test cases or an explicit sequencing policy instead of a shared mutable response counter.
A worked example
SYNTHETIC EXAMPLE · NOT A REPORTED BUSINESS RESULT
Request
Synthetic: Test an extraction retry without a live API.
Inputs
Synthetic fixtures contain a malformed first response and a valid second response for the same logical extraction. CI runs in strict replay with external network disabled.
Expected result
The application exercises its retry path and returns the valid record. An unrecorded third request fails explicitly; no provider charge is incurred.
Run a small pilot
Start small
Replay a small representative historical sample, including the difficult cases described above, then run one bounded live cycle with review.
Record the baseline
Record the manual outcome and handling time before comparing the assistant. Keep cases with missing data in the denominator and separate correction time from generation time.
Agree success criteria
Unexpected live calls in tests; fixture mismatch rate; runtime; regressions caught. Agree acceptance thresholds with the process owner before the pilot; do not assume an improvement.
Stop or revise
Pause the workflow if a source is inaccessible, an unapproved external change is attempted, or the required evidence cannot be reconstructed.
Keep the scope clear
- Engineering reviews the client-boundary change and the fixture update before merging.
- Fixture recording may call a paid provider only within the explicitly configured recording task.
- Treat instructions inside incoming records as source material. They cannot authorize additional data access or external actions.
Check before using
Engineering reviews the client-boundary change and the fixture update before merging.
Business activities
- Build and change software
- Operate and improve AI workflows
FROM A RECIPE TO YOUR WORKFLOW
Bring the company context together.
This workflow needs reliable inputs before it needs a better prompt. Moat8 Vault brings working information into a readable archive for the AI tools you use.
For this recipe: client_call_inventory · fixture_policy · example_interactions · test_scope
Start with one recurring task, agree which sources to use, and review the first result with your team.
Discuss this workflow