Multilingual IVR (Conversational Voice Bot) with Automatic Language Detection

In addition to the aforementioned Amazon Nova 2 Sonic, I recommend also incorporating Strands Agents' BidiAgent.

Reference site:

Quickstart | Strands Agents

【AWS】I built a lightning-fast translation AI agent using StrandsAgents' Bidi agent. #Next.js - Qiita

Yes, this scenario is fully achievable using AWS services. Based on your requirements for automatic language detection and dynamic language switching within the same conversation, I recommend the following approach:

Recommended Solution: Amazon Nova 2 Sonic

Amazon Nova 2 Sonic is specifically designed for this use case and provides native support for automatic language detection and switching within conversations. Key capabilities that address your requirements include:

• Automatic Language Detection and Switching: The model automatically detects the language being spoken and switches languages dynamically during the conversation without requiring menu selection or keypad input.

• Polyglot Voices: Individual voices can speak multiple languages fluidly within a single interaction, maintaining consistent voice characteristics even when languages change mid-conversation.

• Advanced Code-Switching: The model naturally handles mixed-language sentences and can respond in the user's preferred language when they switch from one turn to the next in the same conversational dialog.

• Supported Languages: English (US, UK, India, Australia), French, Italian, German, Spanish, Portuguese, and Hindi. Note that Arabic is not currently listed among the supported languages.

• Telephony Integration: Direct integration with leading telephony providers for phone-based IVR interactions, with support for 8KHz telephony speech input.

• Real-time Streaming: Bidirectional streaming API enables low-latency, multi-turn conversations with intelligent turn-taking and graceful handling of user interruptions.

Architecture Approach:

You can access Amazon Nova 2 Sonic through Amazon Bedrock and integrate it with your telephony infrastructure. The model handles the entire speech-to-speech pipeline, including automatic language detection, speech understanding, and response generation in the detected language.

Important Consideration:

While Nova 2 Sonic supports multiple languages, Arabic is not currently among the supported languages. If Arabic support is essential for your use case, you may need to wait for expanded language support or consider alternative architectures.

Alternative Approach (if Arabic support is required):

If you need Arabic support immediately, you could consider Amazon Lex combined with Amazon Transcribe's multi-language identification feature. Amazon Transcribe can identify and transcribe multiple languages in streaming audio, including when speakers change languages mid-conversation. However, this would require additional integration work to coordinate language detection, natural language understanding, and text-to-speech response generation across multiple services.

The Nova 2 Sonic approach is the most streamlined solution for your requirements, provided the supported languages meet your needs.
Sources
Speech-to-Speech (Amazon Nova 2 Sonic) - Amazon Nova
Introducing Amazon Nova 2 Sonic: Our new speech-to-speech model for conversational AI | AWS News Blog
Language identification with streaming transcriptions - Amazon Transcribe
Amazon Lex – Migrate interactive voice response (IVR) flows to Amazon Lex – Amazon Web Services