Improve a prompt against a defined evaluation set
Iterate on a production task’s prompt using labeled examples and an explicit quality metric instead of judging each rewrite by intuition.
How this works
Iterate on a production task’s prompt using labeled examples and an explicit quality metric instead of judging each rewrite by intuition. Use this when a repeated production task has a measurable quality problem and prompt changes can be compared on representative examples. The workflow should improve the tested task without overfitting to a few attractive demonstrations.
Freeze a baseline, split examples into development and held-out sets, and evaluate candidate prompts under comparable conditions. Inspect error types alongside aggregate scores. Keep model and tool changes separate where possible so improvements can be attributed to the prompt revision.
Before you start
Task definition
Specify the expected output and meaningful failure modes.
Labeled examples
Obtain representative inputs and reviewed expected behavior.
Evaluation harness
Use available scripts or a structured manual comparison with fixed criteria.
Prepare the inputs
- baseline_prompt
- Required
versioned instructions
Preserve the exact production baseline.
- examples
- Required
inputs, expected behavior and coverage labels
Separate development and held-out cases before optimization.
- scoring_rubric
- Required
criteria, weights and hard failures
Use task-relevant measures rather than style preference alone.
- run_configuration
- Required
model, tools and sampling settings
Record conditions that affect comparability.
Implementation, step by step
Measure the baseline
- Run the current prompt on the defined evaluation set.
- Capture outputs, failures and cost or latency if relevant to the decision.
Checkpoint: The starting performance is documented.
Analyze error patterns
- Group errors by missing facts, wrong structure, unsupported conclusions or task-specific failures.
- Inspect examples rather than optimize only an aggregate score.
Checkpoint: Candidate changes target actual weaknesses.
Draft prompt variants
- Make bounded revisions addressing identified error patterns.
- Avoid embedding held-out answers or adding unrelated instructions.
Checkpoint: Each variant has a stated hypothesis.
Compare candidates
- Run variants under the same configuration and inspect scored outputs.
- Review regressions and hard failures, not only average gains.
Checkpoint: A candidate has evidence for or against selection.
Validate held-out behavior
- Evaluate the selected prompt on untouched cases and representative negative triggers.
- Document remaining limitations and prepare a reversible release proposal.
Checkpoint: The owner can judge generalization within the tested scope.
Decisions along the way
Average score improves but a critical case fails
Keep the failure visible and reject or revise according to the agreed rule.
Model changes during testing
Rerun comparable baselines or separate the experiment.
Held-out cases influence revisions
Create a fresh holdout before claiming independent validation.
What you will produce
prompt-candidate.md
MarkdownA reviewed proposed instruction set.
Include: prompt · version · change_hypothesis · limits
prompt-evaluation-report.json
JSONA reproducible comparison with error evidence.
Include: baseline · candidates · case_results · heldout_results · configuration
Check the result
- Verify the holdout was not used to draft candidates.
- Inspect sampled passes and all critical failures.
- Confirm scoring reflects the real task and output contract.
- Check baseline and candidate configurations are comparable.
When something goes wrong
Judge scores are inconsistent
Use reviewed examples and human adjudication for disputed cases.
Evaluation tool fails
Separate infrastructure errors and rerun affected cases.
Prompt grows into unrelated policy
Remove instructions unsupported by task failures.
A worked example
SYNTHETIC EXAMPLE · NOT A REPORTED BUSINESS RESULT
Request
Synthetic: improve invoice-field extraction.
Inputs
Development errors confuse issue and due dates. A candidate explicitly distinguishes them. On a held-out invoice it extracts both correctly but omits currency.
Expected result
Synthetic result: report the date improvement and currency regression separately. Do not select the prompt solely because the development date score rose.
Run a small pilot
Start small
Optimize one stable task with a modest reviewed evaluation set.
Record the baseline
Record baseline errors and manual correction effort.
Agree success criteria
Agree hard-failure and held-out acceptance criteria before selection.
Stop or revise
Stop release recommendations when leakage or incomparable configurations invalidate the test.
Keep the scope clear
- Higher evaluation scores apply only to the tested task and coverage.
- Prompt preparation does not authorize a production change.
Check before using
The source reports production use of iterative prompt optimization, without publishing its benchmark. Hold out test cases, check data leakage, and treat higher evaluation scores as evidence about the tested task only.
Business activities
- Operate and improve AI workflows
FROM A RECIPE TO YOUR WORKFLOW
Bring the company context together.
This workflow needs reliable inputs before it needs a better prompt. Moat8 Vault brings working information into a readable archive for the AI tools you use.
For this recipe: Current task prompt · Representative examples with reviewed expected answers · Quality metric and evaluation procedure
Start with one recurring task, agree which sources to use, and review the first result with your team.
Discuss this workflow