How to Manipulate Chat Responses in RetrieveAndGenerateStream on Amazon Bedrock?

Question

Hello! I am currently using RetrieveAndGenerateStream on Amazon Bedrock with Claude Haiku 3.5. I am facing some issues with manipulating how the chat answers. I tried using the text prompt template with the generation configuration property that you can set in the call, according to this documentation: [https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/bedrock-agent-runtime/command/RetrieveAndGenerateStreamCommand/]. However, whenever I use it, it doesn't retrieve the citations anymore. I have realized that the best responses come from API calls that are cleaner. However, I still need to modulate the answers to the pattern I want. Is there a way to do that without using the text prompt template? If the only possible way to do this is through the generation configuration and text prompt template, what is the correct way to do it?

Alex T · Answer

One likely cause for not seeing citations is not including the $output_format_instructions$ placeholder in your custom prompt template, which provides instructions on how the model should format citations to be picked up by the API. You can find more information about this and the other available placeholders in the "Knowledge base prompt templates: orchestration & generation" section on this page in the Bedrock user guide.

Of course, depending on your custom template it's still possible to come across unreliable citation generation even when the format instructions are there... But including the placeholder should help a lot.

Of course, depending on your custom template it's still *possible* to come across unreliable citation generation even when the format instructions are there... But including the placeholder should help a lot.

How to Manipulate Chat Responses in RetrieveAndGenerateStream on Amazon Bedrock?

Add your answer

Relevant content