Chatbot via Kendra/Bedrock/LangChain returning non-relevant documents

Question

I am building a RAG chatbot on internal corporate documents. My specific architecture is Amazon Kendra for document retrieval, Amazon Bedrock using foundation models (llama or cohere), and LangChain as the orchestrator. I have purposely asked it a question that is not in the corporate documentation. The bot correctly returns that it doesn't know, but it is still returning documents from Kendra. I’m using AmazonKendraRetriever as the retriever and ConversationalRetrievalChain as the chain. Trying to figure out how to not return any src documents when it’s an out of scope question. Any help is appreciated.

zeekg · Answer

Hi Martin, there could be a few factors that result in Kendra returning result based on your query, but from reading your question the issue seems to be the query that is being passed into Kendra. Since Langchain's ConversationalRetrievalChain is designed to rephrase your query utilizing the chat history thats passed in, I would ensure you start with fresh without any chat history and perhaps take a look at the rephrased query that is being sent to Kendra.  Hopefully this a good starting point to troubleshoot the issue. Please feel free to follow up as needed.

Chatbot via Kendra/Bedrock/LangChain returning non-relevant documents

Relevant content