How to allow LLM response so be 20 pages long for QnA chatbot like implementations

Question

Hello,

Currently I have an LLM chatbot running on the sample code from streamlit via this workshop:
https://aws.amazon.com/blogs/machine-learning/quickly-build-high-accuracy-generative-ai-applications-on-enterprise-data-using-amazon-kendra-langchain-and-large-language-models/

and using bedrock with RAG approach using this workshop as sample:
https://github.com/aws-samples/amazon-bedrock-workshop/blob/main/03_QuestionAnswering/01_qa_w_rag_claude.ipynb

I want to query the LLM to write a 20-page user manual from technical specifications sourced via the WebCrawler Kendra Connectors.

However, the current response is cut off after a certain length. What is the bottleneck in causing the cut off of the response? How can I fix it?

Answer

Hi,

Antropic's Claude has currently a limit of a few thousands token on output. But, it can take very large input (up to a 200-page book) .

See https://aws.amazon.com/about-aws/whats-new/2023/08/claude-2-foundation-model-anthropic-amazon-bedrock/

```
Claude 2 can take up to 100,000 tokens in each prompt, meaning it can work over hundreds of 
pages of text, or even an entire book. Claude 2 can also write longer documents—on the order 
of a few thousand tokens—compared to its prior version, giving you even greater ways to develop 
generative AI applications using Amazon Bedrock.
```

To overcome this limitation, your chatbot has to make a sequence of multiple queries to Claude and then you concatenate the answers to present them to the users: "write a documentation about feature X" +  "write a documentation about feature X"  + etc.

Best,

Didier

How to allow LLM response so be 20 pages long for QnA chatbot like implementations

Relevant content