Nova Pro Bedrock Prompt Cache Usage

Question

Hi,

I’ve been trying to use the Prompt Caching feature in NovaPro ( with bedrock api) , but I haven’t been able to get it to work successfully.

I’ve reviewed the documentation and resources on the official website, but I’m still unclear about how to correctly use prompt caching—specifically for input text and system text.

https://aws.amazon.com/blogs/machine-learning/effectively-use-prompt-caching-on-amazon-bedrock/
https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html

Could you kindly provide some guidance or examples on:

How to properly structure input text and system text when setting up prompt caching?
How can I verify that prompt caching is actually working as expected?

Here is the code:

```
    message = {
        "role": "user",
        "content": [
            {"text": input_text},
            {"cachePoint": {"type": "default"}}  
        ]
    }
    
    start_time = time.time()
    response = bedrock_client.converse(
            modelId=model_id,
            messages=[message],
            system=system_text,
            inferenceConfig={
                "maxTokens": 2000,
                "temperature": 0,
                "topP": 0.0
            }
        )
        
```

I’d really appreciate any help or clarification you can provide.

Thank you!

re:Post Agent · Answer

Thank you for your question about using prompt caching with Amazon Bedrock's Nova Pro model. I'd be happy to provide some guidance on how to properly structure your input and system text for prompt caching, as well as how to verify that it's working as expected.

First, let's look at how to structure your input text and system text for prompt caching:

1. Input Text: Your current structure for the input text is correct. You've placed the cache checkpoint after the input text, which is the right approach. This allows the input text to be cached for potential reuse in future requests.

2. System Text: In your current code, you're not applying caching to the system text. To cache the system text, you should structure it similarly to how you've done the input text. Here's how you can modify your code to cache the system text:

```python
system = [
    {"text": system_text},
    {"cachePoint": {"type": "default"}}
]

response = bedrock_client.converse(
    modelId=model_id,
    messages=[message],
    system=system,
    inferenceConfig={
        "maxTokens": 2000,
        "temperature": 0,
        "topP": 0.0
    }
)
```

This change will allow the system text to be cached as well.

To verify that prompt caching is working as expected, you can check the 'usage' section in the API response. This section provides cache performance data, including cache hit rates and token consumption for both read and write operations. Here's how you can modify your code to extract and print this information:

```python
if 'usage' in response:
    usage = response['usage']
    print(f"Cache Read Input Tokens: {usage.get('cacheReadInputTokens', 0)}")
    print(f"Cache Write Input Tokens: {usage.get('cacheWriteInputTokens', 0)}")
    print(f"Input Tokens: {usage.get('inputTokens', 0)}")
    print(f"Output Tokens: {usage.get('outputTokens', 0)}")
```

When prompt caching is working, you should see non-zero values for 'cacheReadInputTokens' in subsequent calls with the same input, indicating that tokens are being read from the cache.

Remember that the cache has a five-minute Time To Live (TTL), so identical requests within this timeframe are more likely to result in cache hits. Also, to fully benefit from caching, try to structure your prompts with static portions that can be reused across multiple requests.

Lastly, keep in mind that while cached tokens are charged at a reduced rate compared to new tokens, there may be additional costs for tokens written to the cache when using certain models.

If you're still not seeing the expected caching behavior after making these changes, you may want to double-check that prompt caching is enabled for your account and the specific model you're using, as it might be subject to preview or regional availability.  
**Sources**  
[Clarification on Prompt Caching Usage with converse API in Amazon Bedrock | AWS re:Post](https://repost.aws/questions/QUIuMZAYVBQHuVYl6tMu82rQ/clarification-on-prompt-caching-usage-with-converse-api-in-amazon-bedrock)  
[Prompt caching for faster model inference - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html)  
[Using the Converse API - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-call.html)  
[Reduce costs and latency with Amazon Bedrock Intelligent Prompt Routing and prompt caching (preview) | AWS News Blog](https://aws.amazon.com/blogs/aws/reduce-costs-and-latency-with-amazon-bedrock-intelligent-prompt-routing-and-prompt-caching-preview/)

Nova Pro Bedrock Prompt Cache Usage

Relevant content