Review incoming workflow examples with a consistent rubric
Screen a large collection of submitted examples for clarity, evidence, and reproducibility before deeper editorial review.
How this works
Screen a large collection of submitted examples for clarity, evidence, and reproducibility before deeper editorial review. A workflow submission can sound impressive while omitting the inputs, review steps, or evidence needed to understand it. A consistent editorial rubric should expose those gaps without pretending that reading a description proves the automation runs.
Screen submissions against explicit criteria for clarity, evidence, reproducibility, and human involvement. Extract supporting passages and unresolved questions. Editors make inclusion decisions; the model's score is a triage aid rather than a verdict on the author or implementation.
Before you start
Editorial rubric
Define criteria and examples of acceptable evidence.
Submission scope
Identify eligible business processes and excluded material.
Review capacity
Set how borderline cases and missing details are handled.
Prepare the inputs
- submissions
- Required
Text with stable IDs
Preserve original wording and attachments metadata.
- rubric
- Required
Criteria and rating definitions
Avoid ambiguous scales.
- eligibility_rules
- Required
Scope conditions
Distinguish relevance from quality.
- evidence_types
- Required
Allowed labels
Separate template, demonstration, and practitioner report.
- review_history
- Required
Prior decisions
Use for consistency checks, not automatic imitation.
Implementation, step by step
Check basic eligibility
- Identify the business task, intended user, and whether the submission fits the collection.
- Keep scope rejection distinct from an assessment of implementation quality.
Checkpoint: An eligibility record.
Extract the workflow structure
- Find the trigger, inputs, transformation, output, tools, and review point.
- Mark omitted elements explicitly rather than completing them from imagination.
Checkpoint: A structured submission summary.
Assess evidence quality
- Identify what was actually observed or reported and the available supporting material.
- Do not label a product description as a verified deployment.
Checkpoint: An evidence classification.
Apply the rubric consistently
- Rate criteria using the defined anchors and cite the relevant passages.
- Surface borderline cases and contradictions instead of forcing a precise overall score.
Checkpoint: A traceable review draft.
Prepare editorial questions
- Ask only for details that would materially change inclusion or interpretation.
- Keep questions specific to the missing input, output, evidence, or boundary.
Checkpoint: A focused clarification list.
Record the editor's decision
- Save accepted, revise, investigate, or excluded status with reasons.
- Use disagreements to refine the rubric in a new version.
Checkpoint: A maintained editorial queue.
Decisions along the way
Workflow is useful but evidence is thin
Keep it as an idea or request details rather than call it tested.
Submission includes private identifiers
Redact the public adaptation while preserving authorized private provenance.
Two reviewers disagree
Compare criterion evidence and resolve the rubric interpretation.
What you will produce
submission-review.csv
CSVSupports consistent editorial triage.
Include: submission_id · eligibility · criterion · rating · evidence · gap · decision
clarification-requests.md
MarkdownPrepares precise follow-up drafts.
Include: submission_id · missing_detail · why_it_matters · question
Check the result
- Verify ratings cite actual submission text.
- Check missing details were not invented.
- Keep evidence type separate from editorial usefulness.
- Review a sample of borderline and rejected submissions.
When something goes wrong
Scores drift across a batch
Recalibrate on shared examples and recheck affected items.
The model rewards polished marketing language
Require concrete process and evidence fields.
An adaptation exposes confidential details
Remove identifiers before any public draft is prepared.
A worked example
SYNTHETIC EXAMPLE · NOT A REPORTED BUSINESS RESULT
Request
Screen two automation examples for a business library.
Inputs
Synthetic example. Synthetic S1 names a trigger, input, reviewed output, and demonstration; S2 says saves hours with no process detail.
Expected result
Synthetic expected result. S1 advances to editorial review without being called independently tested. S2 receives questions about the actual workflow and evidence, not an invented ROI score.
Run a small pilot
Start small
A mixed set of detailed, vague, and borderline submissions.
Record the baseline
Record editor time and disagreement on manual reviews.
Agree success criteria
Agree rubric consistency and useful gap detection.
Stop or revise
Stop batch decisions when rating definitions are unclear or drift materially.
Keep the scope clear
- A review result does not authorize contacting submitters.
- Do not infer the author's competence or intent from submission quality.
Check before using
A model score does not verify that a workflow works. Do not confuse a well-written submission with independently reproduced evidence.
Business activities
- Research customers and markets
- Operate and improve AI workflows
FROM A RECIPE TO YOUR WORKFLOW
Bring the company context together.
This workflow needs reliable inputs before it needs a better prompt. Moat8 Vault brings working information into a readable archive for the AI tools you use.
For this recipe: Submitted workflow descriptions · Review rubric and weights · Examples of strong and weak evidence
Start with one recurring task, agree which sources to use, and review the first result with your team.
Discuss this workflow