Evaluate sales and support agent conversations in batches
A product team used an external workflow to inspect historical conversations while core development continued. Separate evaluation stages made it easier to change criteria and rerun the analysis.
How this works
A product team used an external workflow to inspect historical conversations while core development continued. Separate evaluation stages made it easier to change criteria and rerun the analysis. Use this to inspect historical conversations for specific sales or support failures while keeping the product team focused on development. The evaluation should produce traceable findings and distinguish agent behavior from routing, data or human-handoff problems.
Separate valid customer conversations from spam and tests, identify participant roles, and apply role-specific checks in stages. Keep raw evidence and structured findings linked by conversation ID. Review classification errors before using aggregate results to change the product.
Before you start
Permitted conversation sample
Use records approved for evaluation with unnecessary personal data removed.
Role definitions
Define sales agent, support agent, human operator and customer roles.
Evaluation rubric
Agree task-specific criteria, hard failures and review process.
Prepare the inputs
- dialogues
- Required
conversation ID, turns, timestamps and role metadata
Preserve sequence and mark missing segments.
- sampling_scope
- Required
period, channels and inclusion rules
Do not imply the sample covers all customers.
- role_rubrics
- Required
criteria and evidence requirements per role
Avoid applying sales-conversion criteria to support resolution.
- reference_context
- Required
policies or product facts valid at the time
Use historical applicability where needed.
Implementation, step by step
Validate the sample
- Classify spam, tests, incomplete and eligible conversations.
- Retain exclusions with reasons and review a sample of each class.
Checkpoint: The evaluation population is explicit.
Identify roles and stages
- Map speakers and conversation stages using metadata and evidence.
- Mark uncertain role attribution rather than force it.
Checkpoint: Each dialogue has a usable structure.
Run role-specific checks
- Evaluate factual answers, next steps, escalation and relevant process criteria.
- Attach exact turn references to every finding.
Checkpoint: Results remain traceable.
Inspect errors and patterns
- Review false positives and missed issues manually.
- Aggregate only after classification and rubric corrections.
Checkpoint: Patterns reflect checked evidence.
Prepare product actions
- Summarize recurring gaps and propose bounded fixes or further tests.
- Keep correlation, inferred cause and verified failure distinct.
Checkpoint: The team has actionable findings without unsupported claims.
Decisions along the way
Conversation is incomplete
Assess only visible segments and avoid a final-resolution judgment.
Human operator caused the issue
Attribute the finding to the correct role or handoff.
Criterion is ambiguous
Revise the rubric and rerun affected cases consistently.
What you will produce
dialogue-findings.csv
CSVStructured evidence for each finding.
Include: conversation_id · role · criterion · turn_refs · result · review_status
agent-dialogue-review.md
MarkdownA product review based on checked conversations.
Include: coverage · patterns · examples · limitations · recommendations
Check the result
- Sample inclusion and role classifications for errors.
- Verify findings against the cited turns.
- Confirm aggregate counts use unique eligible conversations.
- Check private content is minimized and access remains restricted.
When something goes wrong
Role metadata is missing
Use cautious evidence-based labeling and retain unknowns.
Rubric change alters results
Version it and rerun the affected batch.
Model judge invents a failure
Remove the unsupported finding and inspect similar cases.
A worked example
SYNTHETIC EXAMPLE · NOT A REPORTED BUSINESS RESULT
Request
Synthetic: evaluate ten support conversations.
Inputs
Two are test scripts, one is truncated, and seven are complete. In C4 the agent promises a refund although policy requires human approval.
Expected result
Synthetic result: exclude the two tests, limit conclusions for the truncated case, and flag C4 with the exact promise and policy reference. Do not report a failure rate over all ten without defining the denominator.
Run a small pilot
Start small
Evaluate a small reviewed batch across sales and support roles.
Record the baseline
Record manual evaluation time and classification errors.
Agree success criteria
Agree finding precision and critical-issue recall before larger batches.
Stop or revise
Stop aggregate interpretation when sampling or role classification is unreliable.
Keep the scope clear
- Conversation evaluation does not authorize contacting customers or changing their records.
- Findings support product review and must not become unsupported individual performance judgments.
Check before using
Review classification errors and source conversations before changing the product. The described workflow used manual spreadsheet batches, with privacy controls still unresolved in the source.
Business activities
- Run sales conversations and proposals
- Resolve customer questions
- Operate and improve AI workflows
FROM A RECIPE TO YOUR WORKFLOW
Bring the company context together.
This workflow needs reliable inputs before it needs a better prompt. Moat8 Vault brings working information into a readable archive for the AI tools you use.
For this recipe: Conversation export · Sales and support evaluation criteria
Start with one recurring task, agree which sources to use, and review the first result with your team.
Discuss this workflow