Skip to content

High Response Latency and Robotic Voice Quality in Amazon Connect Voice Bot Using Amazon Lex, Polly, and Bedrock Integration

0

We have deployed an AI-driven voice bot within our Amazon Connect contact center to handle after-hours (out-of-office) support interactions. Below is a summary of our current architecture:

  1. Amazon Connect routes inbound calls to an out-of-office contact flow.
  2. The flow uses a Get Customer Input block integrated with Amazon Lex for natural language interaction.
  3. The Lex bot includes a QnA intent that invokes an AWS Lambda function when the customer speaks.
  4. The Lambda function integrates with Amazon Bedrock (Titan LLM) to generate dynamic answers using contextual data retrieved from Pinecone (vector database).
  5. The generated response is returned to Lex, which plays it back via Amazon Polly (TTS).
  6. The conversation continues through Elicit Slot prompts until the customer ends the call.

We’re currently facing two major challenges:

1. Unnatural Voice Quality (Amazon Polly)

Despite using Polly Neural TTS voices, the speech still sounds robotic and lacks natural prosody, emotion, and conversational flow, especially when compared to providers like Eleven Labs or Retell AI.

We’re looking for guidance on how to achieve more human-like, conversational voices natively within the Connect + Polly ecosystem. Specifically:

2. High Response Latency (5+ seconds)

We’re observing over 5 seconds of delay between customer speech and AI-generated voice playback. The delay seems to occur during the Bedrock → Lambda → Lex → Polly response chain.

We’d appreciate best practices for achieving real-time or low-latency conversational flow, ideally using streaming integration between Bedrock, Lex, and Connect. In particular:

  • Recommendations for using Lex V2 streaming, Kinesis media streaming, or Bedrock streaming APIs to reduce end-to-end latency.
  • Architectural guidance to support near real-time response similar to third-party voice AI solutions.

**Summary of Assistance Requested **

  1. Recommended AWS-native architecture for minimizing voice response latency in Connect-Lex-Bedrock workflows.
  2. Options for enhancing Polly voice output for natural, human-like conversations.
  3. Any roadmap or preview features (e.g., Nova Sonic, Bedrock streaming) relevant to this use case.
  4. Are there any advanced neural voice models, custom voice fonts, or conversational neural voice capabilities available (or planned) for use within Connect?
  5. Is there any way to integrate Amazon Nova Sonic (or similar high-fidelity voices) into Connect for this use case?
  6. Our goal is to achieve a low-latency, natural-sounding voice bot fully built within the AWS ecosystem, comparable to modern real-time voice AI platforms.

Thank you in advance for your guidance and recommendations.

2 Answers
0

Based on your description of the challenges with your Amazon Connect voice bot, I can provide some recommendations for both the voice quality and latency issues.

Voice Quality Improvements

For the robotic-sounding voice issue with Amazon Polly Neural TTS:

  1. Verify Neural Engine Settings: Ensure that the engine parameter is explicitly set to "neural" in your voice settings. The default engine is "standard" if not specified, which could explain why you're still getting robotic-sounding speech despite selecting a neural voice.

  2. Use SSML Tags for Special Words: For words that need to sound more natural, consider using SSML tags with phonetic pronunciation. This can help with specific words that might otherwise sound robotic even when using Neural TTS.

  3. Voice Selection in Connect: Make sure your Amazon Connect instance is configured to "Use the best available voice" in your instance settings, and verify that your "Set Voice" block settings are properly configured to work with Lex V2.

Latency Reduction Strategies

To address the 5+ second delay in your voice bot responses:

  1. Minimize Conversation Latency: Use latency-optimized inference for foundation models like Amazon Nova Pro to maintain natural conversation flow.

  2. Select Efficient Foundation Models: Consider using smaller, faster foundation models that can deliver quick responses while maintaining quality.

  3. Implement Prompt Caching: Utilize prompt caching to optimize for both speed and cost efficiency, especially in complex scenarios requiring knowledge retrieval.

  4. Deploy TTS Fillers: Use natural filler phrases (such as "Let me look that up for you") before intensive operations to maintain user engagement while the system processes longer-running calls to your foundation models.

  5. Proper Versioning and Aliases: Avoid using the "testbotalias" in production. Create proper versions and aliases for your Lex bot to improve speed and reduce delay.

  6. Architectural Considerations: The current delay is likely occurring in the Bedrock → Lambda → Lex → Polly chain. For real-time voice interactions, consider:

  • Using Amazon Connect Contact Lens real-time streaming or Kinesis Video Streams for accessing live voice
  • Implementing Amazon Transcribe Streaming for real-time speech-to-text conversion
  • Setting up a lightweight WebSocket or API Gateway layer that stays open throughout the call for faster round trips
  1. Region Selection: Consider the trade-off between advanced capabilities like low-latency features and prompt caching (which may only be available in certain regions) versus selecting a region geographically closer to your end-users.

For truly real-time two-way bot conversations, you might want to explore Amazon Connect's External Voice Transfer feature, which allows Connect to SIP conference the call with your bot, letting your bot handle the media while Connect maintains control of the call.

By implementing these recommendations, you should be able to achieve a more natural-sounding voice bot with reduced latency, improving the overall customer experience.
Sources
Voice in Amazon Lex still sounds robotic and delayed even after setting Joanna (NTTS) | AWS re:Post
Building intelligent AI voice agents with Pipecat and Amazon Bedrock – Part 1 | Artificial Intelligence
Is it possible to get real-time voice from a call with just a lambda? | AWS re:Post

answered a year ago

0

Please try Generative voices in Polly - they are the most natural ones in the offering: https://docs.aws.amazon.com/polly/latest/dg/generative-voices.html

AWS

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.