En utilisant AWS re:Post, vous acceptez les AWS re:Post Conditions d’utilisation
AWS re:Postre:Post

Claude V2 api read operation time out exception.

0

I am getting a consistent error of - The read operation timed out.
AWSHTTPSConnectionPool(host='bedrock-runtime.us-east-1.amazonaws.com', port=443): Read timed out. (read timeout=60). This occurs after 5 mts when the document text which I am sending exceeds 20K words. On trying with different values to max_tokens_to_sample parameter, api is throwing throttle exception. Any leads on how to proceed? Thanks in advance.

Sujets
Machine Learning et AIL’IA générative sur AWS
Balises
Amazon BedrockL’IA générative sur AWS
Langue
English
Abhishek
demandé il y a 5 mois547 vues
1 réponse
0

Hi,

You write "v2", did you try with v2, ie. v2.0 or v2.1? I would suggest to try with 2.1 as it is specifically adapted to large inputs (to 200k tokens)

Also, you can try to initially adjust prompt parameters like temperature, top_k, top_p to reduce Claude's creativity: that may lead to less operations when inferring. So higher chance to not time out.

Additionally, I'd try to start with less than 20k words in my prompt and increase incrementally to see where the limit is.

Best,

Didier

profile pictureAWS
EXPERT
Didier_Durand
répondu il y a 5 mois
  • Abhishek
    il y a 5 mois

    I am getting response if using Claude V2 for relatively shorter tokens(lets say 12K words). body= json.dumps({ "prompt": f"\n\nHuman: {usercontent}\n\nAssistant: {assistant}", "max_tokens_to_sample": 2000, "temperature": 0.3, "top_p": 1, })

    modelId = 'anthropic.claude-v2' contentType= 'application/json' accept= 'application/json'

    However, if I am replacing the modelId with 'anthropic.claude-v2:1'. I am getting empty result for shorter tokens and time out for bigger ones.

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions

Contenus pertinents