[Potential Bug] Amazon Multi-Agent Collaboration Routing

Question

I am trying out Amazon Bedrock Multi-Agent Collaboration. While testing, the Supervisor agent was having issues with one of the Collaboration Agents. It detects the Collaboration Agent correctly the first time, but failed to route in the subsequent messages. While debugging, I noticed the following in the Routing trace of the Supervisor Agent:

The last most specialized agent in the conversation is AddUser. Route to this agent using <a>keep_previous_agent</a> if the last user message pertains to a follow up that originated in that agent and that agent requires information from the message to proceed." } ], } ], }, }, "output": { "outputContentType": "application/json", "outputBodyJson": { "content": [ { "type": "text", "text": "<a>keep_previous_agent</a>" } ], },

The output generated is <a>keep_previous_agent</a> instead of AddUser. It looks like the keep_previous_agent is not replaced correctly. However, this happens only when AddUser Collaboration agent is involved and not for other Collaboration agents. Might be because of LLM hallucination (usedClaude 3 Haiku and 3.5 Sonnet). I overrode the Collaboration routing prompt template to use $last_most_specialized_agent$ instead of keep_previous_agent, and its now routing correctly.

re:Post Agent · Answer

Thank you for bringing this issue to our attention. What you've described does indeed sound like a potential bug in the Amazon Bedrock Multi-Agent Collaboration system, specifically related to the routing behavior of the Supervisor agent.

The behavior you're seeing, where the Supervisor agent correctly detects the Collaboration Agent (AddUser) the first time but fails to route subsequent messages properly, is not the expected functionality. The fact that this issue occurs only with the AddUser Collaboration agent and not with others suggests that it might be a specific edge case rather than a general problem with the routing mechanism.

The routing trace you provided shows that the system is correctly identifying the last most specialized agent (AddUser), but then it's failing to properly replace the placeholder <a>keep_previous_agent</a> with the actual agent name. This could indeed be due to a hallucination or processing error in the language model (Claude 3 Haiku or 3.5 Sonnet in this case).

Your workaround of overriding the Collaboration routing prompt template to use $last_most_specialized_agent$ instead of keep_previous_agent is a good temporary solution. This approach directly inserts the name of the last specialized agent, bypassing the potential issue with the placeholder replacement.

Given that this appears to be a specific issue with the routing mechanism, I would recommend the following steps:

Document the exact steps to reproduce this issue, including the configurations of your Supervisor and Collaboration agents.
Collect logs and traces from both successful routings and the problematic ones with the AddUser agent.
Report this as a potential bug to AWS support or through the appropriate channels for Amazon Bedrock feedback. Include your detailed reproduction steps, logs, and the workaround you've implemented.
In the meantime, continue using your workaround with the $last_most_specialized_agent$ placeholder to ensure proper routing.
Keep an eye on updates to Amazon Bedrock, particularly those related to Multi-Agent Collaboration or the specific language models you're using, as a fix may be released in future versions.

This kind of feedback is valuable for improving the service, and AWS will likely be interested in investigating and resolving this potential bug. Thank you for your diligence in identifying and sharing this issue.
Sources
Create multi-agent collaboration - Amazon Bedrock
Maintaining Agent Session Memory in Bedrock Prompt Flow | AWS re:Post

[Potential Bug] Amazon Multi-Agent Collaboration Routing

Add your answer

Relevant content