AI scoring best practices

View summary

AI Scoring evaluates interactions only from the conversation transcript and the evaluation form that you provide.

AI Scoring does not use metadata such as:

• Queue or routing path
• Time stamps or duration
• Platform or system metrics

AI Scoring focuses purely on what was said or written, not on tone, attitude, or inferred meaning.

Do

• Write objective, transcript-driven questions.
• Provide clear help text with measurable criteria.
• Use consistent terminology; for example, “agent” and “customer.”
• Write in complete sentences.
• Refine questions continuously based on AI scoring results.

Don’t

• Ask subjective questions; for example, “Was the agent polite?”
• Use vague shorthand; for example, “Greeting protocol.”
• Assume context not visible in the transcript.
• Ask about tone or emotion that AI cannot infer.

Genesys recommends that each question include short, measurable help text (≤500 characters) that defines what counts as Yes or No, includes examples, and clarifies edge cases.

Example: well-written question and help text

Intent: Verify the customer’s identity before providing account-specific details.

Question: Did the agent verify the customer’s identity before providing account-specific details?

Help text: The agent must confirm at least one credential; for example, account number, date of birth, phone number, or order ID.

• Yes: The agent verified a credential before proceeding.
• No: The agent skipped or asked unrelated questions.

Transcript examples

Establishing alignment between human evaluators and Virtual Supervisor is essential for producing reliable, scalable quality evaluations. One effective approach is to have an expert human evaluator manually score a subset of interactions to establish an agreement rate—a measure of how closely human evaluations align with the AI-generated scores produced by Virtual Supervisor.

A high agreement rate indicates that both human reviewers and the AI are consistently interpreting and applying the evaluation criteria. A lower agreement rate can point to differences in interpretation, unclear scoring instructions, or evaluation questions that require refinement. By analyzing where discrepancies occur, teams can improve question wording and scoring guidance, ensuring that both human and AI evaluations reflect the intended standards and can be applied consistently at scale.

How to establish and improve agreement rates

1. Create a new evaluation form
   • Click Menu > Conversation Intelligence > Quality Management > Evaluation Forms.
   • Click Create and then select Agent Auto-Evaluation as the form type.
   • Build the evaluation form using the best practices outlined in this article.
   • Publish the form.
2. Evaluate interactions using the form
   • Click Menu > Analytics > Analytics Workspace > Interactions.
   • Open an individual interaction.
   • In the Quality Summary tab, click Create Evaluation.
   • Select the Agent Auto-Complete evaluation form.
   • Choose one of the following options:
     • For automated submission, select Virtual Supervisor as the evaluator.
     • To review automated scores before submission, select a human evaluator.
   • Click Create. If you select a human evaluator, review the completed evaluation and update any answers that were incorrectly selected based on transcript evidence.
3. Test the form across multiple interactions:
   1. Repeat the evaluation process for at least 20 different interactions.
   2. Click Conversation Intelligence > Quality Management > Evaluation Forms and open the tested form.
   3. Review the overall agreement rate and the agreement rates for individual questions.
4. Refine the form as needed:
   • Use the agreement data to identify questions or scoring instructions that need adjustment.
   • Update the evaluation form and repeat the testing process until acceptable agreement levels are achieved.

• Review AI scoring reports monthly to identify low-confidence questions.
• Add examples reflecting your industry’s phrasing; for example, healthcare vs. retail.
• Align human evaluators with the same definitions used by AI.
• Refine questions and help text when AI misclassifies results.

Question:
Did the agent greet the customer at the beginning of the conversation using a standard greeting such as “hi,” “hello,” or “good morning”?

Answer Options:
Yes / No

Help Text:
The agent must begin with a polite greeting. Acceptable greetings include “hi,” “hello,” “good morning,” or equivalent.

Transcript Example:
Customer: “I need help with my account.”
Agent: “Good morning! Thank you for calling ABC Support.”
→ AI marks Yes

Question:
Did the agent verify the customer’s identity before addressing account-specific concerns?

Answer Options:
Yes / No

Help Text:
The agent must verify at least one customer credential, such as date of birth, account ID, or phone number, before proceeding.

Transcript Example:
Customer: “I want to update my billing address.”
Agent: “Can I confirm the last four digits of your account number?”
→ AI marks Yes

Question:
Did the agent allow the customer to finish speaking without interruptions?

Answer Options:
Yes / No

Help Text:
An interruption occurs when the agent cuts the customer off mid-sentence. Clarifications after the customer finishes are acceptable.

Transcript Example:
Customer: “The issue started when—”
Agent: “Let me stop you there…”
→ AI marks No.

Question:
Did the agent use the customer’s name at least once during the conversation?

Answer Options:
Yes / No

Help Text:
Using the customer’s name personalizes the interaction. The AI checks whether the name, as provided in the record or conversation, was mentioned.

Transcript Example:
Customer: “I’m Maria, and I need help with my order.”
Agent: “Thank you, Maria. Let’s look at your order.”
→ AI marks Yes

Question:
Did the agent acknowledge the customer’s issue before moving to resolution?

Answer Options:
Yes / No

Help Text:
Empathy is demonstrated when the agent acknowledges the issue with phrases such as “I understand,” “I’m sorry that happened,” or “I know this must be frustrating.” The AI looks for acknowledgment before resolution.

Transcript Example:
Customer: “I’ve been overcharged.”
Agent: “I’m sorry to hear that. Let’s review your bill together.”
→ AI marks Yes

Question:
Did the agent provide a clear resolution step, such as a refund, replacement, or troubleshooting instruction?

Answer Options:
Yes / No

Help Text:
A resolution must include a specific action (refund, replacement, or technical steps). General apologies without clear next steps do not count.

Transcript Example:
Agent: “I’ve processed a replacement order that will arrive within two days.”
→ AI marks Yes

Question:
Did the agent explain the escalation process clearly, including who to contact, what information is needed, and expected response times?

Answer Options:
Yes / No

Help Text:
A complete escalation includes:

1. Next contact point.
2. Required information.
3. Expected timeframe for response.

Transcript Example:
Agent: “If this issue persists, I will escalate to Level 2. They will need your case ID and will respond within 24 hours.”
→ AI marks Yes

Question:
Did the agent comply with mandatory disclosure or compliance statements (for example, terms, disclaimers, or legal requirements)?

Answer Options:
Yes / No

Help Text:
Compliance statements vary by industry and may include required disclosures, disclaimers, or legal notices. AI checks for these specific phrases in the transcript.

Transcript Example:
Agent: “For security purposes, I cannot provide account details without verifying your identity first.”
→ AI marks Yes

Question:
Did the agent avoid unnecessary dead air or long silences without explanation?

Answer Options:
Yes / No

Help Text:
Silences longer than 15 seconds are acceptable only when explained to the customer (for example, “I will place you on a brief hold while I check this”).

Transcript Example:
Agent: “Give me a moment while I pull up your account details.” [10 seconds of silence]
→ AI marks Yes

Question:
Did the agent confirm customer satisfaction or summarize the next steps before closing the conversation?

Answer Options:
Yes / No

Help Text:
Before closing, the agent should confirm the resolution or clearly summarize the next steps. This path demonstrates completion and helps ensure customer understanding.

Transcript Example:
Agent: “I’ve reset your password. Can you confirm you’re able to log in now?”
→ AI marks Yes