By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Amazon Bedrock Claude V2 InvokeModelRequest Body Size Limit

1

I made a demo project in C# using the AWS SDK 3.3 testing Claude V2. I gave it a research document to summarize. It seems that when the body of the InvokeModelRequest reaches around 154K bytes (the document is around 20K words), I get the following exception: {"The request was aborted: The request was canceled."} {"Cannot close stream until all bytes are written."}

I thought that Claude could handle up to 100K tokens, which I am definitely under.

  • Abhishek
    4 months ago

    I too am getting a consistent error of - The read operation timed out. AWSHTTPSConnectionPool(host='bedrock-runtime.us-east-1.amazonaws.com', port=443): Read timed out. (read timeout=60). This occurs when the document text which I am sending exceeds 20K words. On trying with different values to max_tokens_to_sample parameter, api is throwing throttle exception. Any leads on how to proceed? Thanks in advance.

  • technomage
    2 months ago

    I'm also seeing performance way below the indicated 100k/200k token limits. Using word count as an approximate token proxy, I can get maybe 20k words maximum.

    I get request timeouts with a body size of around 200k characters.

Topics
Machine Learning & AIGenerative AI on AWS
Tags
Machine Learning & AIGenerative AI on AWS
Language
English
ckb2023
asked 7 months ago1351 views
2 Answers
0

Have you already checked this AWS documentation. As per this documentation:

Enter image description here

Hope this helps.

Abhishek

profile pictureAWS
EXPERT
secondabhi_aws
answered 7 months ago
  • ckb2023
    7 months ago

    I thought "max_tokens_to_sample" only affected the response. Anyway, I played around with that parameter and set it all the way to 30K, in which my query actually worked. It's odd since the documentation says the limit is 8K. I could not set it above 30K however.

0

Also having this issue. When we support more than about 10k tokens we get "An error occurred (ValidationException) when calling the InvokeModel operation: Input is too long for requested model.". Well below the 100k token limit we get through Anthropic's API. Also getting an error when posting the same request in the playground. I think something's actually broken in Bedrock

liams_s
answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content