Problems using langchain and Bedrock with titan FM

0

I'm struggling to understand what's going on here, so just asking for insights or help :)

I'm trying to implement RAG using a Faiss vector store on nodejs running against the titan express FM.

However the results I'm getting are not dependable in the simple case, and non-existent when trying to submit context with my query. If I directly invoke the model I usually get a sensible result but even that often gives "Sorry - this model is unable to respond to this request." with exactly the same query:

$ node index-4.js
Query Amazon Bedrock:
question: 'what does article 25 of the gdpr say?'
Direct invocation: Article 25 of the GDPR states that the data subject has the right to request the erasure of his or her personal data from the controller without undue delay. The controller shall have the right to determine the time limits for erasure.
Individual steps: Sorry - this model is unable to respond to this request.
Use VectorDBAQChain: Sorry - this model is unable to respond to this request.

$ node index-4.js
Query Amazon Bedrock:
question: 'what does article 25 of the gdpr say?'
Direct invocation: Sorry - this model is unable to respond to this request.
Individual steps: Sorry - this model is unable to respond to this request.
Use VectorDBAQChain: Sorry - this model is unable to respond to this request.

Any suggestions for how I can debug this code? I don't know what's flakey here, langchain or the FM:

async function queryBedrock(vs_filename, docs, question)
{   
    const model = new Bedrock({
        model:"amazon.titan-text-express-v1",
        region: process.env.BEDROCK_REGION,
        credentials: {
            accessKeyId: process.env.AWS_ACCESS_KEY_ID,
            secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
        },
        temperature: 0.7,
        maxTokens: 2500
    });

    /////////////////////////////
    // invoke model directly
    /////////////////////////////

    try {
        let result = await model.invoke(question);
        console.log(`Direct invocation: ${result}`);
    }
    catch (e){
        console.log(e);
    }

    /////////////////////////////
    // use individual steps
    /////////////////////////////

    const embeddings = new BedrockEmbeddings({
        model: "amazon.titan-embed-text-v1",
        region: process.env.BEDROCK_REGION,
        credentials: {
            accessKeyId: process.env.AWS_ACCESS_KEY_ID,
            secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
        },
    });
    
    let vectorStore = await FaissStore.load(vs_filename, embeddings);
    const retriever = vectorStore.asRetriever();
    const prompt = PromptTemplate.fromTemplate(`Answer the question based only on the following context: {context} Question: {question}`);
    const indiv_chain = RunnableSequence.from([
                    {   
                        context: retriever.pipe(formatDocumentsAsString),
                        question: new RunnablePassthrough(),
                    },
                    prompt,
                    model,
                    new StringOutputParser(),
                ]);
    
    const result = await indiv_chain.invoke(question);
    console.log(`Individual steps: ${result}`);

    /////////////////////////////
    // use VectorDBQAChain
    /////////////////////////////

    const dbqa_chain = VectorDBQAChain.fromLLM(
                        model,
                        vectorStore,
                        { k: 10, returnSourceDocuments: true }
                    );

    const res = await dbqa_chain.call({
        input_documents: docs,
        query: question
    });
    console.log(`Use VectorDBAQChain: ${result}`);
}
dmb0058
asked 5 months ago765 views
2 Answers
0

Enable logging on Bedrock in order to see which prompts are being sent to the model. This will allow you to determine if the problem lies in the retrieval, eg the context passed in the prompt is not existent or wrong, or in the model. Once you have the prompts, try to execute them directly to see if there is flakiness or if they are systematically returning no answers. You can also try to manually modify such prompts in order to try to get the answers and then make the necessary adjustments to your solution.

AWS
EXPERT
answered 5 months ago
0

Great idea! I'm only starting with Bedrock and disn't realise it had logging so this is really useful thanks.

I've established that it is highly variable (same prompt may or may not give a result) and extremely prompt-sensitive (slight change of context such as "... in no fewer than 450 words ..." vs "... in no fewer than 500 words ..." makes a significant difference) compared to OpenAI, so I've managed to get some good output but your tips will help me to figure out what's working and what isn't.

dmb0058
answered 5 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions