Does Nova Sonic use audio chunks or ASR text for reasoning when generating responses and function calls?

Hi Leo, thanks so much for posting your question in re:Post.

Based on the AWS documentation, Nova Sonic does understand prosody and tone from the raw audio, not just ASR text. Here's what the documentation confirms:

Nova Sonic's Audio Understanding Capabilities

From the AWS AI Service Card for Nova Sonic:

"Amazon Nova Sonic can recognize and semantically understand speech input... Intrinsic variation refers to features of the speech input to which the model should attend, e.g. prosodic elements such as intonation that conveys different meaning or a sense of urgency."

This explicitly states that Nova Sonic:

1. Attends to prosodic elements (tone, intonation, delivery)
2. Extracts semantic meaning from these audio features
3. Uses prosody to understand different meanings and urgency

Additionally, the documentation mentions:

"Adaptive speech response that dynamically adjusts delivery based on the prosody of the input speech."

This confirms Nova Sonic processes the actual audio characteristics, not just transcribed text.

How It Works

Nova Sonic is a unified speech-to-speech model, meaning:

• It processes audio directly (not ASR → LLM → TTS pipeline)
• The model understands both linguistic content AND paralinguistic features (tone, emotion, prosody)
• Function calls and reasoning have access to this audio-derived understanding

Why Your Tone Evaluation May Feel Inconsistent

Here are considerations for the specific usage characteristics you provided:

1. Function Call Context

When Nova Sonic calls your rateCandidateTone function, it passes text parameters, not audio features. The model's internal understanding of prosody influences its reasoning, but the function call itself only receives text-based information.

2. Confounding Variation

The documentation notes:

"Confounding variation refers to features of the input that the model should ignore, e.g., variations in acoustic conditions including background noise"

Nova Sonic is designed to be robust to noise, which means it may filter out some acoustic features you consider important for tone evaluation.

3. Model Focus

Nova Sonic prioritizes:

• Semantic understanding (what was said)
• Conversational flow (natural dialogue)
• Response generation (appropriate replies)

Detailed tone/emotion analysis may not be its primary strength, even though it uses prosody for understanding.

Recommendations

Option 1: Enhance Your System Prompt

Guide Nova Sonic to pay more attention to emotional/tonal aspects:

When evaluating the speaker's tone, pay close attention to:
- Speaking pace and rhythm
- Vocal intensity and energy
- Hesitations or confidence in delivery
- Emotional undertones (enthusiasm, frustration, uncertainty)

When calling rateCandidateTone, provide detailed observations about these vocal characteristics.

I hope this helps with your application!

Option 2: Pre-Process Audio Separately

For more reliable tone analysis, consider:

• Using a dedicated speech emotion recognition (SER) model alongside Nova Sonic
• AWS services like Amazon Transcribe with sentiment analysis
• Third-party emotion detection APIs that analyze raw audio

Option 3: Capture Tone in Conversation

Instead of relying solely on the final function call, have Nova Sonic:

• Make observations about tone throughout the conversation
• Store these observations in conversation history
• Aggregate them when calling rateCandidateTone

Example:

System prompt: "Throughout the conversation, mentally note the speaker's tone, confidence level, and communication style. When asked to rate their tone, synthesize these observations."

Documentation References

• Nova Sonic Overview
• Nova Sonic AI Service Card - See "Intended use cases and limitations"
• Output Events - Shows ASR transcription flow

Summary

Yes, Nova Sonic uses audio chunks (prosody, tone) for reasoning, not just ASR text. However, the function call interface is text-based, and the model's tone understanding may not be granular enough for detailed communication style evaluation. Consider supplementing with explicit prompting or dedicated audio analysis tools for more consistent results.