Amazon Nova Sonic - Intercept Response

To route responses from Amazon Nova Sonic to your frontend as data/actions instead of speech, you can leverage the event-driven architecture of Nova Sonic's bidirectional stream API.

Here's how you can implement this approach:

1. Use the tool use capability: When you want Nova Sonic to return data/actions instead of speech, you can design your prompts to instruct the model to use specific tools or functions. Nova Sonic will send tool-related events with specific tool names and parameters that your frontend can intercept and process.

2. Monitor event types in the response stream: Nova Sonic's responses include multiple content blocks in sequence:

• ASR transcription (what the user said)
• Tool use events (when the model invokes functions)
• Text response (preview of planned speech)
• Audio response (the actual speech)

3. Intercept specific events: Your application can listen for and process specific event types:

• For speech responses: Use the audio response events containing base64-encoded speech chunks
• For data/actions: Use the tool use events or text response events with specific markers

4. Implement conditional handling: In your frontend code, add logic to:

• Process tool use events to extract structured data
• Render UI components or trigger actions based on the data
• Optionally suppress audio playback for responses you want to handle as data

5. Design your prompts accordingly: Structure your system prompts to instruct Nova Sonic when to use speech versus when to use tools for returning data.

This approach allows you to maintain a single conversation stream while selectively determining which responses should be spoken to the user and which should be processed as data actions in your frontend application.
Sources
Make your web apps hands-free with Amazon Nova Sonic | Artificial Intelligence
Handling output events with the bidirectional API - Amazon Nova
Using the Amazon Nova Sonic Speech-to-Speech model - Amazon Nova