Bedrock Tools - Token Usage
I am evaluating leveraging tools via the Bedrock Converse API. One thing I was hoping to use it for was to guarantee a certain JSON format in the model's response. I am currently providing formatting instructions in the prompt which seems to work OK in most cases, but not guaranteed.
I wasn't able to find any information in the docs about how using tools affects the input and output token counts for a given request.
I tested two similar requests:
- Minimal prompt and a simple tool for formatting the output, temperature 0
- Same base prompt but added formatting requirements to prompt, no tools, temperature 0
The request with the tool resulted in a drastically higher input and output token count. Based on this I'm assuming the toolConfig in the request counts towards the input token count and the entire output object in the response (which for tool requests includes more data) counts towards the output token count. Can you confirm and provide any more detail?
- Tags
- Amazon Bedrock
- Language
- English
- Newest
- Most votes
- Most comments
Hi Jeff,
Documentation [1] indicate that when using tools, the toolConfig in the request counts towards the input token count, and the entire output object in the response (which includes more data when using tools) counts towards the output token count.
Please check the AWS Community article "Generating JSON with the Amazon Bedrock Converse API" [2] which answers your question in two parts
- Quick tutorial on Amazon Bedrock Converse API
- How to use tools with Converse API
[1] https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html [2] https://community.aws/content/2hWA16FSt2bIzKs0Z1fgJBwu589/generating-json-with-the-amazon-bedrock-converse-api [3] https://community.aws/content/2hW5367isgQOkkXLYjp4JB3Pe16/intro-to-tool-use-with-the-amazon-bedrock-converse-api
Hope this helps
Relevant content
- AWS OFFICIALUpdated 2 months ago
Thanks @vatsravi for your response. I actually don't see any reference to token usage as it relates to toolConfig in the referenced documentation & article, but your answer aligns with what I observed. I think this should be clearly stated in the documentation.