Bedrock Converse API - Consistent Timeout Issue with Larger models like Anthropic Sonnet 3.5 and Meta Llama 405B

Question

And yes it couldn't be any sort of rate-limit because it was my first message in last 48hrs.

Region Name - us-west-2

```
WARNING:bedrock_converse.utils:Retrying core.llama.bedrock_converse.utils.converse_with_retry_async._retry in 4.0 seconds as it raised ReadTimeoutError: Read timeout on endpoint URL: "https://bedrock-runtime.us-west-2.amazonaws.com/model/meta.llama3-1-405b-instruct-v1%3A0/converse".
Retrying bedrock_converse.utils.converse_with_retry_async._retry in 4.0 seconds as it raised ReadTimeoutError: Read timeout on endpoint URL: "https://bedrock-runtime.us-west-2.amazonaws.com/model/meta.llama3-1-405b-instruct-v1%3A0/converse"
```

Michael Shapira · Answer

Try to add this. Note that it is client side timeout and not server-side.

```
 config = Config(
     connect_timeout=300,
     read_timeout=300,
 )
 bedrock_client = boto3.client(service_name='bedrock-runtime', config=config)
```

Bedrock Converse API - Consistent Timeout Issue with Larger models like Anthropic Sonnet 3.5 and Meta Llama 405B

Add your answer

Relevant content