How do I prevent read timeouts in Python when I use large models in Amazon Bedrock?

Short description

Sometimes when you use Amazon Bedrock's large language models to generate text you get read timeout errors. These errors occur when the Python boto3 client queries the large language model but doesn't receive a response within botocore's default read timeout period. To resolve read timeout errors, increase the read timeout

Resolution

Example code to increase the read timeout value for the client:

from boto3 import client
from botocore.config import Config

config = Config(read_timeout=1000)

client = client(service_name='bedrock-runtime',
                      config=config)

If you use third-party libraries such as LangChain, first instantiate a boto3 client with a botocore configuration. Then, pass this as a client parameter to a callable model class.

Example code to increase the read timeout value when passing a boto3 client to third-party libraries.

from boto3 import client
from botocore.config import Config
from langchain_community.llms import Bedrock

config = Config(read_timeout=1000)

client = client(service_name='bedrock-runtime',
                      config=config)

llm = Bedrock(model_id="anthropic.claude-v2:1",
              client=client)

The preceding examples show that the read timeout is set to 1000 seconds. The read timeout period specifies how long botocore waits for a response from the server before it throws a read timeout exception.
Note: Large models such as Anthropic Claude Opus can take more than sixty seconds to return a response.

Best practices

Set the read timeout to be long enough to allow your queries to complete. Start with a large value such as 1000 seconds, and then adjust this duration until you no longer get a timeout error.