Connect GenAI evaluation Form

Hello, thanks for the question. Please see the best practices to provide instructions using GenAI performance evaluations as per this link: https://docs.aws.amazon.com/connect/latest/adminguide/generative-ai-performance-evaluations.html#guidelines-to-improve-generative-ai-accuracy

1. Improving phrasing of questions and associated instructions
2. Use complete sentences to word questions, for example, replacing ID validation with "Did the agent attempt to validate the customer’s identity?" enables the generative AI to better understand the question.
3. It is recommended that you provide detailed criteria for answering the question within the instructions to evaluators, especially if its not possible to answer the question based on the question text alone. For example, for the question "Did the agent try to validate the customer identity?" you may want to provide additional instructions such as, The agent is required to always ask a customer their membership ID and postal code before addressing the customer’s questions.
4. If answering a question requires knowledge of some business specific terms, then specify those terms in the instruction. For example, if the agent needs to specify the name of the department in the greeting, then list the required department name(s) that the agent needs to state as part of the instructions to evaluators associated with the question.
5. If possible, use the term 'agent' instead of terms like 'colleague', 'employee', 'representative', 'advocate', or 'associate'. Similarly use the term 'customer', instead of terms like 'member', 'caller', 'guest', or 'subscriber'.
6. Only use double quotes in your instruction if you want to check for exact words being spoken by the agent or the customer. For example, If the instruction is to check for the agent saying "Have a nice day", then the generative AI will not detect Have a nice afternoon. Instead the instruction should say: The agent wished the customer a nice day.

When using generative AI for agent performance evaluations in Amazon Connect, providing clear instructions to evaluators is crucial as these instructions are also used by the AI to power automated evaluations. Here are some best practices:

1. Use complete sentences when phrasing evaluation questions and clearly specify the evaluation criteria within the instructions to improve the accuracy of generative AI evaluations.

2. Provide specific criteria for answering questions in the instructions to evaluators for each evaluation form question. This not only drives consistency among human evaluators but also helps the generative AI interpret and analyze conversation transcripts more accurately.

3. Focus on questions that can be answered using information from conversation transcripts, as generative AI works best with objective questions that can be verified from the transcript data.

4. Avoid highly subjective or numeric questions for automated evaluations, as these may not be suitable for generative AI analysis.

5. Consider the phrasing of your questions to improve AI accuracy. Be specific about what you're looking for in agent performance.

6. When setting up automated evaluations, ensure your instructions clearly outline what constitutes successful performance for each question, allowing the AI to make appropriate assessments.

7. Use the calibration session feature to test and refine your evaluation form. After finalizing a calibration session, you can download a report that shows how different evaluators (including AI) scored the same interaction, which can help you identify questions that get inconsistent answers and may need improvement in phrasing or specificity.

By following these practices, you can maximize the effectiveness of generative AI-powered evaluations in Amazon Connect while maintaining consistency in your agent performance assessment process.
Sources
Evaluate agent performance in Amazon Connect using generative AI - Amazon Connect
Create an evaluation form in Amazon Connect - Amazon Connect
Calibration sessions for performance evaluations - Amazon Connect