Skip to content

Amazon Bedrock Knowledge Base evaluation jobs failing silently

0

I'm encountering an issue with Amazon Bedrock Knowledge Base evaluation jobs failing silently. Here are the specific details:

Issue:

  • Created a knowledge base evaluation job using AWS Management Console
  • Evaluation type: Retrieval and Response generation
  • Evaluator model: Claude 3.5 Sonnet
  • Quality metrics: Helpfulness and Harmfulness (default metrics)
  • Job status changes from "In Progress" to "Failed" after a few minutes
  • No failure reason is displayed when clicking on the Failed status. Says No failure reason available
  • S3 output folder only contains an empty file named "amazon-bedrock-evaluations-permission-check"
  • No relevant logs found in CloudWatch Log Insights

Sample evaluation dataset structure:

{
    "conversationTurns": [{
        "referenceResponses": [{
            "content": [{
                "text": "This is a reference context"
            }]
        }],
        "prompt": {
            "content": [{
                "text": "This is a prompt"
            }]
        }
    }]
}

Questions:

  1. How can I identify the root cause of the evaluation job failure?
  2. Are there specific CloudWatch log groups/streams I should check for Bedrock evaluation job logs?
  3. What are the common causes for knowledge base evaluation jobs failing without error messages?
  4. Are there any known limitations or requirements for the evaluation dataset format?

Any guidance on troubleshooting this issue would be greatly appreciated.

1 Answer
1

To address your issue with Amazon Bedrock Knowledge Base evaluation jobs failing silently, I can provide some guidance on troubleshooting and potential causes:

  1. To identify the root cause of the evaluation job failure:

    • Double-check the IAM permissions associated with the role used for the evaluation job. Ensure it has the necessary permissions for Amazon Bedrock, S3, and any other related services.
    • Verify that the service role has proper access to Amazon OpenSearch Serverless collections if you're using them.
    • Check if the data access policy for any collections includes the service role under the Principal section.
    • Ensure that Amazon Bedrock is correctly authenticated with AWS Secrets Manager if it's being used.
  2. For CloudWatch logs specific to Bedrock evaluation jobs: Unfortunately, I don't have specific information about dedicated CloudWatch log groups or streams for Bedrock evaluation jobs. You may need to contact AWS support for more detailed logging information.

  3. Common causes for knowledge base evaluation jobs failing without error messages:

    • Insufficient permissions or misconfigured IAM roles
    • Issues with the underlying data sources or storage configurations
    • Problems with the evaluation dataset format or content
    • Temporary service issues or resource constraints
  4. Regarding limitations or requirements for the evaluation dataset format: Your sample evaluation dataset structure appears to be in the correct format. However, ensure that:

    • The file format is supported by Bedrock Knowledge Base (e.g., JSON, CSV, TXT)
    • The dataset adheres to any size limitations (though I don't have specific information on size limits)
    • All required fields are present and correctly formatted in your dataset

Additional troubleshooting steps:

  • Try creating a simpler evaluation job with fewer metrics or a smaller dataset to isolate the issue.
  • Verify that the S3 bucket permissions allow Bedrock to write evaluation results.
  • Check if there are any service quotas or limits you might be hitting.
  • Ensure that the evaluator model (Claude 3.5 Sonnet) is supported and available in your region for knowledge base evaluations.

If the issue persists after checking these points, you may need to contact AWS support for more in-depth troubleshooting, as they can access logs and details not available through the console or API.
Sources
Resolve Amazon Bedrock knowledge base permission errors | AWS re:Post
Issue related to AWS Bedrock Knowledge Base - File format not supported | AWS re:Post
Choose the best performing knowledge base using Amazon Bedrock evaluations - Amazon Bedrock
Creating a knowledge base evaluation job in Amazon Bedrock - Amazon Bedrock

answered 2 years ago

EXPERT

reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.