Conversation reviews

Review transcripts, check topic citations, and diagnose agent behavior.

Lesson 6 of 6 — Review transcripts and understand agent behavior.

Check transcripts and matched topics to confirm the agent behaved as expected.

What conversation review is for (level 1)

At this stage, use Conversation Review to answer simple but critical questions:

Understanding

Did the agent understand the user?

Topic matching

Did it answer from the right Managed Topic?

Escalation

Did it escalate or hand off when expected?

Errors

Did anything obviously go wrong?

You are not tuning behavior yet. You are confirming that the basics work.

Starting in Conversations

Begin on the Conversations page.

This table lists all interactions across Chat and Call.

Focus on these columns:

• Status: Live or Ended
• Summary: High-level intent label
• Start time: Helps align with test runs
• Contact: Phone number or user identifier
• Duration: Longer calls often signal confusion
• PolyScore (if enabled): Quick quality indicator

Use Edit table to add helpful columns:

• Environment
• Variant
• Handoff reason
• Function call

This makes basic issues easier to spot without opening every conversation.

Filtering during testing

Use Filter conversations to narrow your view.

Common Level 1 filters:

• Environment = Sandbox
• Start date = Today
• Duration > 60 seconds
• Status = Ended

Example: Filter to Sandbox + Duration > 90s to find conversations where the agent struggled.

Opening a conversation

Clicking a row opens Conversation Review.

You will see three main areas:

1. Transcript (center)
2. Conversation details (left)
3. Diagnostic toggles (right)

At Level 1, focus primarily on the transcript and matched topics.

Reading the transcript

The transcript shows the conversation turn by turn.

For each turn, check:

• What the user said (ASR text)
• How the agent responded
• Which Managed Topic was matched (shown beneath the turn)

Example: User: “late checkout”

Matched topic: late_checkout_policy

This is the expected outcome. If the wrong topic appears, the issue is almost always in Managed Topic naming or sample questions.

Check your understanding

Matched topics

Matched topics are your primary signal at this level.

Use them to confirm:

• The correct topic was selected
• Similar topics are not competing
• The topic name reflects the intent clearly

Example: The agent answers about billing, but the matched topic is general_help.

This indicates a Managed Topic design problem, not a phrasing issue.

Conversation details (left panel)

The left panel confirms context.

Check:

• Environment (Sandbox vs Live)
• Channel (Chat or Call)
• Language and locale
• Duration
• Variant (if applicable)

This helps ensure you are reviewing the correct test and not a different version of the agent.

Basic diagnosis toggles

At Level 1, you only need a small subset of diagnosis tools.

Topic citations

Use this to confirm which KB topic was used for each response.

If the agent answers correctly but cites the wrong topic, fix the KB before proceeding.

Function calls (preview)

Functions, SMS, and handoffs are covered in Level 2. At Level 1, you likely don’t have any functions yet. If your project does use them, this toggle shows whether actions fired — but you don’t need it for basic diagnosis.

Audio playback (Calls only)

For call channels, use audio playback to:

• Confirm ASR quality
• Hear how long responses feel when spoken
• Check whether users interrupt the agent

Split audio is especially useful for hearing the caller clearly.

Annotations

Annotations let you flag issues directly from the review page.

At Level 1, use:

• Wrong transcription → ASR problem
• Missing topic → KB gap

Annotations help you build a clear list of fixes instead of relying on memory.

What good looks like

After a review session you can say:

• The agent selected the correct KB topic
• Responses matched the questions
• Escalation was reasonable
• Any problems trace to KB or ASR

If you can’t clearly state what to fix, simplify before adding complexity.

Check your understanding

Before level 2

• You can explain each response in plain terms
• You can identify KB vs ASR issues
• The agent behaves consistently in Sandbox

Try it yourself

1. Challenge: Diagnose a mismatch

   A user asked “do you accept dogs?” and the matched topic in Conversation Review shows general_help instead of pet_policy.

   Answer:

   1. What is the most likely cause?
   2. How would you fix it?
   3. How would you verify the fix worked?

   Hint

   The agent responded but used the wrong topic. This is a retrieval problem, not an ASR problem — ASR transcribed the question correctly. Think about what drives topic selection.

   Example solution

   1. Most likely cause: The pet_policy topic’s name is too vague or its sample questions don’t include “dogs”, “pets”, or “animals” — so general_help was a closer match.
   2. Fix: Rename the topic to pet_policy if it isn’t already, and add sample questions like “do you accept dogs”, “are pets allowed”, “can I bring my cat”. Ensure general_help doesn’t have overlapping sample questions.
   3. Verify: In Chat, ask “do you accept dogs?” and check that the topic citation now shows pet_policy. Test several phrasings and confirm the topic is consistent.

Check your understanding

← Previous: Environments & versions

Lesson 5 of 6

Level 1 complete →

Recap and next steps