Build a reusable review rubric from approved meetings
Extract recurring questions from an authorized reviewer’s meeting records, turn them into an anonymous rubric, and test it against held-out examples.
How this works
Use approved conversations to document how a review is conducted, for example the questions used in project diligence or an investment screen. The objective is a reusable set of criteria and question patterns. It is not a biography, a voice clone or a claim that the system can replace the original reviewer.
Keep source transcripts in the permitted workspace. AI groups questions and proposes criteria; the human reviewer decides which patterns are intentional and worth retaining. Use separate material for extraction, iterative calibration and a final untouched test. Report calibration performance separately from final evaluation.
Before you start
Authorized corpus
Confirm which meeting records can be analyzed and what may be retained in the reusable output.
Defined reviewer
Select the role and meeting types whose method is being documented; exclude unrelated conversations.
Evaluation set
Reserve calibration examples and a separate final test set with human judgments before generating the rubric.
Prepare the inputs
- meetings
- Required
Transcripts with speaker IDs, meeting type and time references
Confirm consent and intended scope; exclude sensitive records outside that scope.
- reviewer_scope
- Required
Role and target decision context
Do not infer private characteristics from conversational patterns.
- evaluation_examples
- Required
Held-out documents plus reviewer notes
Keep extraction, calibration and final test sets separate. Never use final-test answers to tune the reported version.
Implementation, step by step
Select and partition
- Inventory eligible meetings and identify where the target reviewer actually led the questioning.
- Set aside calibration and final test material before extraction; record the three-way split without exposing participants.
Checkpoint: The extraction, calibration and untouched final test sets have separate stable IDs.
Extract questions with context
- Capture the question, preceding subject and relevant follow-up using transcript references.
- Separate the reviewer’s questions from answers, quoted statements and another participant’s suggestions.
Checkpoint: Each extracted item is traceable and correctly attributed to a role.
Identify reusable criteria
- Group questions by the uncertainty they are trying to resolve.
- Draft a criterion with when to use it, what evidence it asks for and when it does not apply. Label rare patterns as tentative.
Checkpoint: Every rule has a rationale and an evidence basis.
Remove identifying content
- Replace people, clients, deal names and exact distinctive figures with generic roles or synthetic examples.
- Check combinations of facts that could still reveal the original meeting. Remove unnecessary excerpts rather than merely deleting names.
Checkpoint: The reusable package contains principles without source identities.
Calibrate, then test a frozen version
- Apply the draft to calibration examples and compare questions with the reviewer’s assessment. Record useful questions, missed concerns and unsupported inferences; revise with a documented reason.
- Freeze the rubric version, then evaluate it once on the untouched final set. If those results drive revisions, label the old set calibration material and reserve a new final set.
Checkpoint: Calibration results and untouched final-test results are reported separately.
Version and approve
- Have the reviewer approve criteria, applicability and privacy treatment.
- Save the rubric version, change log, evaluation limits and owner. Keep raw evidence separate from the shareable package.
Checkpoint: Users know which version and scope they are relying on.
Decisions along the way
A pattern appears only once
Retain it as a tentative example until the reviewer confirms its generality.
Speakers cannot be distinguished
Exclude the segment or request clarification; do not assign questions by writing style.
The rubric fails a new type of meeting
Mark that context unsupported and collect appropriate examples before expanding scope.
What you will produce
review-rubric.md
MarkdownAn anonymous reusable review method.
Include: criterion_id · purpose · when_to_use · question_pattern · evidence_needed · exceptions
rubric-calibration.csv
CSVA held-out evaluation record without raw confidential transcripts.
Include: example_id · criterion_id · expected_concern · observed_question · error · reviewer_decision
Check the result
- Verify questions against transcript locations and speaker roles.
- Verify the three-way split and confirm final test examples were not used to tune the reported rubric version.
- Search for original identities and distinctive confidential details in outputs.
- Measure missed concerns and unsupported questions separately.
When something goes wrong
The rubric overfits one transaction
Broaden the permitted sample or reduce the claimed scope.
An example remains identifiable
Replace it with a synthetic example and repeat the privacy review.
Reviewer feedback changes a criterion
Version the change, rerun calibration examples and use new untouched final examples if independent performance is claimed.
A worked example
SYNTHETIC EXAMPLE · NOT A REPORTED BUSINESS RESULT
Request
Synthetic: capture a project-review method.
Inputs
Eight authorized meetings supply question patterns; two projects support calibration and two other projects remain untouched for final evaluation.
Expected result
One criterion asks who accepts the result and what evidence they need. After calibration, a frozen version is tested on the two untouched projects; reports contain no client identities.
Run a small pilot
Start small
Use a small permissioned extraction corpus, calibration examples and a separate final test set from the target meeting type.
Record the baseline
Record the reviewer’s questions and material concerns before showing the AI output.
Agree success criteria
Agree acceptable missed-concern and unsupported-question rates; require a clean privacy review.
Stop or revise
Stop reuse if identities leak, speaker attribution is unreliable or no independent evaluation is available.
Keep the scope clear
- A rubric captures a review method; it does not reproduce a person or authorize decisions.
- Sharing the rubric does not grant access to the source meeting records.
Check before using
The reviewer approves the rubric and privacy check before reuse; it supports judgment rather than replacing it.
Business activities
- Raise capital or evaluate investments
- Maintain shared company knowledge
FROM A RECIPE TO YOUR WORKFLOW
Bring the company context together.
This workflow needs reliable inputs before it needs a better prompt. Moat8 Vault brings working information into a readable archive for the AI tools you use.
For this recipe: Approved meeting transcripts · Reviewer scope · Held-out review examples
Start with one recurring task, agree which sources to use, and review the first result with your team.
Discuss this workflow