- Newest
- Most votes
- Most comments
Same issue here. Unacceptable. AWS Bedrock starting to become a massive annoyance. If I'd self-manage most steps I'd already be done by now.
Anyways, are there any alternative approaches? As of my current project I need to work with AWS Bedrock. So any ideas how to fix this, are greatly welcome.
answered a year ago
When generating text with large language models in AWS Bedrock, you may encounter read timeout errors from the botocore client used to communicate with the Bedrock endpoint. This is because large model queries can take longer to process and return a response, potentially exceeding the default botocore read timeout.
To prevent read timeouts, you can increase the botocore read timeout configuration. Here is an example of increasing the read timeout to 1000 seconds:
from boto3 import client
from botocore.config import Config
config = Config(read_timeout=1000)
client = boto3.client(service_name='bedrock-runtime',
region_name='us-east-1',
config=config)
answered 2 years ago
In the more specific case where the same exact request usually returns successfully in under ten seconds but occasionally times and after 60 seconds before retrying successfully, what is the proper way to diagnose these timeouts? It seems simply setting the timeout to longer would only make you wait longer for the same result. Is that true, or are you expecting a more specific exception? Ten seconds is acceptable for a user-interactive application while 60 seconds is not.
I have tested the Bedrock endpoint in a different region (host='bedrock-runtime.us-west-2.amazonaws.com', port=443) using the exact same input tokens repeatedly in succession, and the responses are very inconsistent. The same request usually finishes in under 10 seconds, but sometimes it is 1 minute and <10 seconds, with nothing in between. This points to an intermittent problem on the Bedrock server side. What would be the next steps in finding the root cause in this case?
I just confirmed that, in my case of connecting to bedrock-runtime.us-west-2.amazonaws.com from an EC2 instance, increasing the timeout from 60 seconds to 300 seconds changes nothing except the amount of time required to detect the read timeout.
Something weird is that calling the same model directly from Anthropic seems to be faster. This is not matter of latency because we call Bedrock API from a different region. It seems that Bedrock takes just longer generating tokens than the original models in Anthropic.
@Joe did you notice the same?
@joe @Amit Lulla Any finding or anyway you were able to fix this issue?
answered 9 months ago
Relevant content
asked 2 years ago
- AWS OFFICIALUpdated a year ago
We couldn't find a solution for this. We are calling Anthropic directly now.