Support Reporting
Part of Support quality tools
Building a fair conversation review sample
Define eligible conversations, draw a baseline sample and separate risk-triggered reviews so quality findings have a clear population.
Build a fair review sample by defining which conversations are eligible, then selecting from that population with a recorded rule.
Keep cases chosen because they appear risky in a separate queue. Otherwise, a review score may mainly describe what reviewers chose to inspect.
Define the population
State the period, channels, teams and unit of review. A team might, for example, review closed email and messaging conversations from one calendar month, counting each conversation once.
Decide in advance how to handle bot starts, several agents, reopened cases and contacts with no customer question.
For coaching, identify the part of a shared conversation the agent could control.
For a service-process review, examine the wider customer journey. Record these as different purposes rather than giving both the same unexplained score.
Draw a baseline and keep exceptions separate
Select baseline cases at random from the eligible population. If channels or issue types differ substantially, select within stated groups and record both the eligible and reviewed counts for each group.
A random draw reduces selection bias on average; it does not guarantee that every small sample reflects the whole population.
A complaint, low customer rating or suspected incorrect answer may warrant prompt review regardless of the draw. Record its trigger and report targeted findings separately.
Do not combine targeted and random cases into one unqualified quality rate.
Register item / Record
- Eligibility
- Dates, channels, status and exclusions
- Selection
- Random method or named risk trigger
- Unit
- Conversation, agent contribution or message
- Coverage
- Eligible and selected counts by relevant group
- Review state
- Completed, pending or excluded with reason
Key Metrics for Review Sampling
- Eligible cases
- Total conversations meeting criteria (e.g., closed email and messaging from one month)
- Reviewed cases
- Number of cases selected for review (random or risk-triggered)
- Baseline sample size
- Randomly selected subset reflecting population distribution
- Targeted reviews
- Cases flagged for risk (e.g., low rating, complaint)
Keep the selection intact
Freeze the selected list for the cycle and record cases reviewers cannot access. Replace one only under a rule agreed before review.
Give reviewers the relevant history and the policy in force when the agent replied.
Ask two reviewers to score a shared set and discuss differences in how they applied the criteria. Clarify the scorecard before treating a score change as a change in service quality.
Product settings need the same scrutiny. Intercom describes Monitors as defining which conversations get reviewed, with criteria, reviewers and a scorecard; its Pro add-on describes their use for routine QA across a baseline sample, high-risk issues or a specific theme or event.
Zendesk QA documents conversation filters and calibration. A filter or Monitor still needs a written eligibility and selection rule.
Report baseline findings with eligible and reviewed counts. If groups had different chances of selection, show them separately or account for those differences before presenting an overall figure.
Report targeted findings as issues needing action, with owners.
