Bedrock ConverseAPI on Lambda using Cachepoint behavior
I have a specific implementation question about using cachepoint in lambda. Below is the scenario
User is using Nova Pro model. User wants to process 35 images in a lambda function and leverage the caching feature. They first initiate a bedrock client. Then initiate converseapi call, put 20 images into it (because of the limit) with a question get a response back. (cachepoint default) Next, they initiate second converseapi call and add the next 15 images. with same question (cache point default) and get response. Now they want to send a 3rd prompt which is different question from the first 2. This question should be answered using the 35 images using converse api. This time they dont include the cachepoint flag in the api call.
They expect final response from lambda execution to be result from the 3 converse calls.
Issue: The 3rd prompt should have used the cached images but cacheread usage shows 0 and the response is unknown. The 3rd prompt results in response not using the cache. Is this the right way to use the cachepoint?
- Language
- English
asked a year ago363 views
- Newest
- Most votes
- Most comments
Based on the scenario you've described, there are a few important points to consider regarding the use of prompt caching with the Amazon Bedrock Converse API in a Lambda function:
-
Caching Behavior: The cache has a five-minute Time To Live (TTL). This means that cached content is only available for a short period after it's initially cached.
-
Cache Checkpoints: For the Nova Pro model, you can set multiple cache checkpoints in a single request. However, the cache is tied to the specific structure and content of the prompt up to the checkpoint.
-
Subsequent Calls: When you make subsequent calls without including a cache checkpoint, the system doesn't automatically use the previously cached content.
Given these points, here's an analysis of your approach:
-
First two calls: Your approach of using cache checkpoints in the first two calls is correct. This should cache the content of the 35 images.
-
Third call: The issue arises in your third call. By not including a cache checkpoint in this call, you're essentially telling the system not to use the cache. This explains why you're seeing 0 for cache read usage.
To correctly leverage the cached content in your third call, you should:
- Structure your third prompt to match the structure of the first two calls up to the cache checkpoint.
- Include the cache checkpoint in your third call, even though you're asking a different question.
- Add your new question after the cache checkpoint.
This approach should allow the system to use the cached content of the 35 images while still processing your new question.
It's also worth noting that the cache is specific to the structure and content of the prompt. If your third call significantly changes the structure or content before the checkpoint, it may not hit the cache even if you include a checkpoint.
To optimize your use of caching:
- Try to keep the structure of your prompts consistent across calls.
- Place cache checkpoints strategically to maximize reuse of cached content.
- Monitor the usage data returned by the API to understand your cache hit rates and adjust your strategy accordingly.
Remember, the effectiveness of caching can vary based on the specific details of your implementation and the nature of your prompts. You may need to experiment with different approaches to find the optimal caching strategy for your use case.
Sources
Clarification on Prompt Caching Usage with converse API in Amazon Bedrock | AWS re:Post
Prompt caching for faster model inference - Amazon Bedrock
Effectively use prompt caching on Amazon Bedrock | AWS Machine Learning Blog
answered a year ago
Relevant content
asked a year ago
asked a year ago
