Bedrock JAVA Converse API cache not working

Question

Hello,

I'm trying to use Bedrock Converse API for Amazon Nova Pro with prompt cache for `message`. I've wrote the below piece of code to build the message, but I did not found a way to enable caching.
```
var message = Message.builder()
                .content(ContentBlock.fromText("some text"))
                .role(ConversationRole.USER)
                .build();
ConverseResponse response = client.converse(request -> request
                    .modelId(modelId)
                    .messages(message)
                    .inferenceConfig(config -> config
                            .maxTokens(512)
                            .temperature(0.5F)
                            .topP(0.9F)));
```
The invocation works, but the `usage` shows that cache is not used: `TokenUsage(InputTokens=2053, OutputTokens=148, TotalTokens=2201)`
It should have looked something like this:
```
  "usage": {
    "input_tokens": 9,
    "cache_creation_input_tokens": 37209,
    "cache_read_input_tokens": 0,
    "output_tokens": 357
  }
```
Also [documentation ](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html) specifies:

> Amazon Nova offers automatic prompt caching for all text prompts, including User and System messages. This mechanism can provide latency benefits when prompts begin with repetitive parts, even without explicit configuration. However, to unlock cost savings and ensure more consistent performance benefits, we recommend opting in to Explicit Prompt Caching.

> If you used prompt caching, then in the usage field,  cacheReadInputTokensCount and cacheWriteInputTokensCount tell you how many total tokens were read from the cache and written to the cache, respectively.

Could you please specify how to use explicit caching in JAVA for above Bedrock, Amazon Nova Pro Foundation Model, Converse API ?

re:Post Agent · Accepted Answer

To enable explicit prompt caching with the Bedrock Converse API in Java for Amazon Nova Pro, you need to add a cache checkpoint to your message content. Your current code doesn't include this cache checkpoint, which is why you're not seeing any cache usage in the response.

Here's how you can modify your code to include a cache checkpoint:

```java
var message = Message.builder()
.role(ConversationRole.USER)
.content(List.of(
ContentBlock.fromText("some text"),
ContentBlock.builder()
.cachePoint(CachePoint.builder()
.type("default")
.build())
.build()
))
.build();

ConverseResponse response = client.converse(request -> request
.modelId(modelId)
.messages(message)
.inferenceConfig(config -> config
.maxTokens(512)
.temperature(0.5F)
.topP(0.9F)));
```

The key change is adding a `cachePoint` ContentBlock after your text content. This tells Bedrock to cache everything before this point.

You can also add cache checkpoints to the system prompt if you're using one:

```java
var systemPrompt = List.of(
ContentBlock.fromText("Your system instructions here"),
ContentBlock.builder()
.cachePoint(CachePoint.builder()
.type("default")
.build())
.build()
);

ConverseResponse response = client.converse(request -> request
.modelId(modelId)
.messages(message)
.system(systemPrompt)
.inferenceConfig(config -> config
.maxTokens(512)
.temperature(0.5F)
.topP(0.9F)));
```

To verify that caching is working, check the `usage` field in the response. When prompt caching is active, you should see values for `cacheReadInputTokens` and `cacheWriteInputTokens`. On the first call, you'll see tokens being written to the cache, and on subsequent identical calls (within the 5-minute TTL period), you should see tokens being read from the cache.

Remember that the cache has a 5-minute Time To Live (TTL), so identical requests need to be made within this timeframe to benefit from cache hits.  
**Sources**  
[Clarification on Prompt Caching Usage with converse API in Amazon Bedrock | AWS re:Post](https://repost.aws/questions/QUIuMZAYVBQHuVYl6tMu82rQ/clarification-on-prompt-caching-usage-with-converse-api-in-amazon-bedrock)  
[Nova Pro Bedrock Prompt Cache Usage | AWS re:Post](https://repost.aws/questions/QUGudA0owsTMS1BtKDMpWQ8w/nova-pro-bedrock-prompt-cache-usage)  
[Using the Converse API - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-call.html)  
[Effectively use prompt caching on Amazon Bedrock | AWS Machine Learning Blog](https://aws.amazon.com/blogs/machine-learning/effectively-use-prompt-caching-on-amazon-bedrock/)

Bedrock JAVA Converse API cache not working

Add your answer

Relevant content