Process audio during a phone call using Connect

Hi!

I'm trying to create a POC using Connect to place an outbound call. Then, I need to be able to capture the customer's audio, convert it to text using Amazon Transcribe, send that text representation to an internal API that uses Bedrock, process the answer, and convert it back to audio using Amazon Polly.

I have enabled data streaming in Connect using a Kinesis stream. I'm able to get the streaming from Kinesis using the MediaStreams and the StartFragmentNumber from KinesisVideo.

My questions are:

1. Is there any way to get the audio from Polly and play it during the same phone call?
2. I have tried different combinations without success. Do you have any suggestions on how to achieve this?
3. Sometimes the streaming is not as near real-time as desired. Is there a way to improve the latency? Any hints will be greatly appreciated.

This loop will continue until the client hangs up or the LLM concludes that the required information has been fulfilled, and therefore the call can be hung up.

Lex was evaluated, but the intents are quite variable, which is why we would like to use an LLM.

Best regards!