Bedrock Agent "rationale" field missing from response when using custom orchestration prompt template

Greeting

Hi Patrick,

Thank you for your detailed question! It's clear that you've put a lot of thought into crafting your custom prompt template for your Bedrock agent. Let’s dive into your issue regarding the missing rationale field and how you can preserve it while maintaining your desired functionality.

---

Clarifying the Issue

From your description, it seems you're overriding the default prompt template in Bedrock's orchestration to achieve consistent structured JSON outputs. While this approach works well for generating the outputs you want, it appears to have removed the rationale field from the orchestration trace. The rationale field, as noted in the AWS documentation, provides chain-of-thought reasoning and is valuable for your UI.

The core issue lies in how the default orchestration logic and metadata generation interact with custom templates. By modifying the orchestration prompt, you've inadvertently changed the system's behavior, leading to the absence of the rationale field in the traces. The solution involves understanding how rationale generation works in Bedrock and updating your custom template to preserve this behavior.

---

Key Terms

• Bedrock Agent: A service for building conversational AI applications, leveraging LLMs with orchestrated function calls.
• Rationale Field: Part of the orchestration trace containing chain-of-thought reasoning, often used for debugging or UI displays.
• Orchestration Trace: A structured log detailing model input, output, metadata, rationale, and observations for each invocation.

---

The Solution (Our Recipe)

Steps at a Glance:

1. Understand how Bedrock generates the rationale field.
2. Modify your custom template to include rationale generation hooks.
3. Test and validate the updated template for trace consistency.
4. Add example orchestration traces to verify behavior before and after the fix.
5. Debug and validate using Bedrock's trace logging and AWS support if necessary.

---

Step-by-Step Guide:

1. Understand Rationale Generation:
   Bedrock generates the rationale field as part of its internal orchestration logic. This happens when the default prompt template includes specific hooks or system instructions that align with the rationale field generation process. Customizing the template may bypass or suppress these hooks, leading to the field’s omission.

---

2. Modify the Custom Template:
   Update your custom orchestration template to ensure it explicitly supports rationale generation. To do this:
   • Include explicit instructions in the system section for the agent to provide a rationale for its actions or chain-of-thought reasoning.
   • For example, append a guideline like this:

     - Provide a detailed rationale for your thought process before creating a plan or invoking functions.

   • Structure the messages section to encourage rationale outputs. For instance:

     {
       "role": "assistant",
       "content": [{"type": "text", "text": "Explain your reasoning before presenting your answer: $agent_scratchpad$"}]
     }

---

3. Test and Validate:
   Deploy the updated template and monitor orchestration traces. Check if the rationale field reappears. You can also cross-reference with the modelInvocationInput and modelInvocationOutput to ensure consistency.

---

4. Add Example Orchestration Trace:
   Including example traces can help validate and understand the changes:

   Before Customization (Missing rationale):

   {
       "modelInvocationInput": { ... },
       "modelInvocationOutput": {
           "metadata": { "usage": { "inputToken": 12, "outputToken": 24 } },
           "rawResponse": { "content": "..." },
           "observation": { "text": "Processed the request." }
       }
   }

   After Fix (With rationale):

   {
       "modelInvocationInput": { ... },
       "modelInvocationOutput": {
           "metadata": { "usage": { "inputToken": 12, "outputToken": 24 } },
           "rawResponse": { "content": "..." },
           "rationale": { "reasoning": "Chain of thought explaining plan." },
           "observation": { "text": "Processed the request." }
       }
   }

---

5. Debug and Validate:
   • Enable detailed trace logging in the Bedrock console to ensure the system captures all fields, including rationale.
   • Review the modelInvocationInput and compare it to the generated output to verify consistency.
   • If needed, contact AWS support to verify whether any internal orchestration logic needs to be re-enabled when using custom templates.

---

Closing Thoughts

The missing rationale field likely stems from how your custom template interacts with Bedrock’s internal orchestration mechanisms. By explicitly including hooks for rationale generation in your custom template, you can maintain your structured JSON outputs while preserving the chain-of-thought data for your UI.

A practical tip: test your updates in a staging environment before moving to production and document your updated template to streamline future adjustments or team collaboration. Let me know how it goes, Patrick!

---

Farewell

I hope this refined guidance helps you restore the rationale field while keeping your custom template functional. Feel free to reach out with any updates or additional questions. Good luck! 😊

---

Cheers,

Aaron 😊