Bedrock LLMs cache

Question

Does Bedrock API's for LLMs have a cache to answer the same input?

Im using claude-instant-v1 model with 0 temperature and I see that sometimes for the same input the response is the same and other ones it changes all of a sudden.

Is there any way that we can ensure that the response will be the same for the same input? 
if im keeping the same model_id, could boto3 version updates have an impact in the bedrock api response?

Didier Durand · Answer

Hi,

See this interesting discussion thread  on why LLM completions are different even with temperature = 0

https://community.openai.com/t/a-question-on-determinism/8185/8

Best
Didier

Behrens, Isaac · Answer

I don't believe you can ever *ensure* the same answer from an LLM as they are not giving canned responses, but rather doing mathematical calculations based on probability of next words based on weights of the association of words in training data.

Here is an article on the topic:  https://cset.georgetown.edu/article/the-surprising-power-of-next-word-prediction-large-language-models-explained-part-1/#:~:text=When%20an%20LLM%20receives%20a,picked%20to%20continue%20the%20text.

Hope this helps!

Bedrock LLMs cache

Add your answer

Relevant content