Decide what to do next with an automation pilot
Use observed pilot runs to choose whether to improve the workflow, share it with colleagues, automate its execution, or keep it limited. Include the time spent checking and correcting outputs.
How this works
Use observed pilot runs to choose whether to improve the workflow, share it with colleagues, automate its execution, or keep it limited. Include the time spent checking and correcting outputs.
Judge a pilot from observed runs, including review and correction effort. Separate successful normal examples from exception behavior and choose a next step proportional to the evidence rather than treating a few good outputs as production readiness.
Before you start
Defined operating scope
An automation pilot has observable runs, correction effort, and defined quality criteria.
Required source material
Obtain Pilot outputs and observations; Quality criteria; Timing, frequency, and maintenance estimates. If connectors are unavailable, use authorized files with the same fields.
Decision and exception rules
Only successful runs were retained. Reconstruct or rerun representative failures before making a scale decision.
Prepare the inputs
- run_log
- Required
Inputs, outputs and run conditions
Keep failures and manual corrections, not only successful examples.
- acceptance_rules
- Required
Checkable quality criteria
Define critical and tolerable errors in advance.
- effort_records
- Required
Preparation, review, correction and maintenance time
Use comparable units and periods.
- decision_options
- Required
Improve, share, automate or limit
State prerequisites for each next step.
Implementation, step by step
Establish the working record
- Confirm the selected records, effective versions and responsible owner.
- Check run_log, acceptance_rules, effort_records, decision_options before beginning. Record the source version and the selected scope; an unavailable value stays unknown rather than becoming an inferred fact.
Checkpoint: The input record identifies the selected run_log and any missing required fields.
Compare a normal run with excess-data and missing-source cases
- Compare ordinary runs with excess-data and missing-source cases, preserving the exact inputs and observed behavior.
- Recompute total effort from all stages and compare like-for-like cases.
Checkpoint: A saved intermediate result for: Compare a normal run with excess-data and missing-source cases.
Assess outputs against explicit, checkable criteria and record manual corrections
- Apply the quality criteria and record human corrections and unresolved failures by type.
- Check the decision cites observed runs and explicitly includes failed exception cases.
Checkpoint: A saved intermediate result for: Assess outputs against explicit, checkable criteria and record manual corrections.
Compare potential time savings with review, maintenance, and error costs
- Compare total assisted handling effort with the baseline, including recurring maintenance and error handling assumptions.
- Recompute total effort from all stages and compare like-for-like cases.
Checkpoint: A saved intermediate result for: Compare potential time savings with review, maintenance, and error costs.
Draft a next-step recommendation that points to the observed evidence
- Draft a recommendation with supporting runs, limitations and the conditions that must be met before the next expansion.
- Check the decision cites observed runs and explicitly includes failed exception cases.
Checkpoint: A saved intermediate result for: Draft a next-step recommendation that points to the observed evidence.
Package the result and unresolved work
- The process owner approves the next step. A few successful trials are an initial screen, not proof of production reliability.
- A small pilot is an initial evidence screen, not proof of reliability.
Checkpoint: A pilot decision brief with its evidence and remaining conditions. Unresolved items remain visible in the review record.
Decisions along the way
Only successful runs were retained.
Reconstruct or rerun representative failures before making a scale decision.
Review effort consumes the apparent preparation saving.
Report the net result honestly and investigate a narrower task.
An exception produces an unsupported consequential answer.
Keep the pilot limited and fix the failure before automating execution.
What you will produce
pilot-runs-to-scale-decision-result.json
JSONA pilot decision brief with its evidence and remaining conditions. Label proposed and completed states separately.
Include: run_id · case_type · quality_result · corrections · total_effort · baseline_comparison · recommendation · conditions · status · evidence_reference · review_note
pilot-runs-to-scale-decision-review.md
MarkdownKeep the evidence and unresolved decisions needed to apply this result, including exceptions involving decision_options.
Include: source_reference · question · owner · next_action
Check the result
- Recompute total effort from all stages and compare like-for-like cases.
- Check the decision cites observed runs and explicitly includes failed exception cases.
- Inspect run_log and acceptance_rules in both the original and output; confirm that record identity and interpretation survived the transformation.
- The process owner approves the next step. A few successful trials are an initial screen, not proof of production reliability.
When something goes wrong
Only successful runs were retained.
Reconstruct or rerun representative failures before making a scale decision. Retain the affected evidence and recovery status.
Review effort consumes the apparent preparation saving.
Report the net result honestly and investigate a narrower task. Retain the affected evidence and recovery status.
An exception produces an unsupported consequential answer.
Keep the pilot limited and fix the failure before automating execution. Retain the affected evidence and recovery status.
A worked example
SYNTHETIC EXAMPLE · NOT A REPORTED BUSINESS RESULT
Request
Decide what to do next with an automation pilot for the following bounded example; prepare the result for review.
Inputs
Synthetic example: Manual handling takes 20 minutes. Pilot preparation takes 5, review 8 and correction 9 on a difficult case; a missing-source run invents a value.
Expected result
Synthetic expected result: Difficult-case total is 22 minutes, two above baseline. Recommend refinement and continued limits, with the fabricated-value failure blocking automated execution.
Run a small pilot
Start small
Trial a representative batch containing the ordinary case and the exception demonstrated above.
Record the baseline
Record current preparation time, review effort and correction types for the same records.
Agree success criteria
Agree acceptable review and error limits; measure net handling time including review; quality failures across normal and exception cases.
Stop or revise
An exception produces an unsupported consequential answer. Pause affected records and apply this recovery: Keep the pilot limited and fix the failure before automating execution.
Keep the scope clear
- A small pilot is an initial evidence screen, not proof of reliability.
- The process owner approves expansion; a recommendation does not authorize unattended execution.
Check before using
The process owner approves the next step. A few successful trials are an initial screen, not proof of production reliability.
Business activities
- Operate and improve AI workflows
FROM A RECIPE TO YOUR WORKFLOW
Bring the company context together.
This workflow needs reliable inputs before it needs a better prompt. Moat8 Vault brings working information into a readable archive for the AI tools you use.
For this recipe: Pilot outputs and observations · Quality criteria · Timing, frequency, and maintenance estimates
Start with one recurring task, agree which sources to use, and review the first result with your team.
Discuss this workflow