ThrottlingException while asynchronously invoking Bedrock Runtime in Lambda


Hey, I am using AWS Bedrock runtime client in AWS Lambda. So in python when I am trying to run multiple requests (6) to the claude2 model asynchronously using Threads, some times I get all the results and a lot of the times I get ThrottlingException saying too many attempts. The thing is, is there a way to avoid this from happening as it happens randomly, with the same input tokens and everything. Is it an infra related problem, also please guide me in fixing this. Thankyou.

1 Answer

You have 2 options to consume a model via Bedrock, On-demand vs Provisioned Throughput.

As per the documentation, latency differs by model and is directly proportional to the following conditions.

  • The number of input and output tokens
  • The total number of ongoing on-demand requests by all customers at the time.

You can purchase Provisioned Throughput to address your issue.

Please have a look at the below thread for a similar issue.

profile pictureAWS
answered 3 months ago
profile pictureAWS
reviewed a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions