Knowledge Base
Part of AI support tools
Testing AI answers against approved support material
The point of an AI answer test is to check whether the response follows the support material customers are meant to rely on.
An AI answer test checks whether the response follows the support material customers are meant to rely on. A fluent answer that invents an exception to a refund or access rule is worse than an honest handoff.
Freeze a small set of approved articles and their versions for the test. Create questions that require a direct answer, a qualification, a clarification and a refusal to guess. Include one question whose answer changed recently.
Compare each generated response with the source text. Did it preserve eligibility, timing and exclusions, or quietly simplify them?
Zendesk describes connected help centres and external knowledge sources for AI-generated replies. It notes that external content may be as old as its last sync.
A changed source page should trigger a freshness check before a tool is judged accurate. This is one product example; it does not prove every support platform behaves the same way.
Keep a review sheet with question, expected source, answer, error type and correction owner. Separate unsupported claims from outdated content and from genuinely ambiguous policy. Only widen the allowed question set after the team can explain failures and correct them reliably.
Run the same questions again after any change to the articles, the prompt or the tool settings. A repeat run shows whether a fix removed the error or only moved it elsewhere. Keep the earlier results next to the new ones so the team can see which answers improved, which got worse and which stayed the same.
Running an AI answer test against approved support material
- Freeze the approved articles and their versions for the test
- Write questions needing a direct answer, a qualification, a clarification and a refusal to guess
- Include one question whose answer changed recently
- Compare each generated response with the source text for eligibility, timing and exclusions
- Trigger a freshness check when a source page has changed before judging the tool accurate
- Log the question, expected source, answer, error type and correction owner in a review sheet
- Widen the allowed question set only once failures can be explained and corrected reliably
Question types to include in an AI answer test
- A question that requires a direct answer
- A question that requires a qualification
- A question that requires a clarification
- A question that requires a refusal to guess
- One question whose answer changed recently
