Amazon Bedrock Knowledge Bases with metadata filtering

Here's one way of doing it through using the Boto3 to access the Bedrock SDK in Python (code is in quotation marks).

import boto3 

bedrock_cli = boto3.client("bedrock", region_name="your-region-name")
bucket_name = "name of bucket"
prefix = " "

metadata_filter = {
      "startsWith": {
             "key": "x-amz-bedrock-kb-source-uri",
             "value": f"s3://{bucket_name}/{prefix}"
       }
}

response = bedrock_cli.retrieve_and_generate(
    knowledgeBaseId="your-knowledge-base-id",  # Replace with your Knowledge Base ID
    inputText="Tell me about Neo Tokyo games.",
    maxResults=5,  # Adjust to your needs
    metadataFilters=[metadata_filter]
)

Make sure your S3 metadata filter matches the filter key. Make sure to tag them correctly when they are uploaded into the S3 bucket.

Hope the code above helps!

Amazon Bedrock Knowledge Bases indeed supports metadata filtering, which can be used to improve retrieval accuracy by narrowing down search results based on specific document attributes. In your case, you're looking to filter based on file names in the S3 bucket that is part of your knowledge base.

The example you provided is correct and can be used to filter documents based on their S3 prefix. This approach leverages the built-in metadata attribute "x-amz-bedrock-kb-source-uri" which represents the S3 URI of the document. By using the "startsWith" operator, you can filter documents that are located in a specific folder within your S3 bucket.

Here's how you can use this filter in the retrieve_and_generate method:

1. First, construct your filter as you've shown:

   publisher_filter = { "startsWith": { "key": "x-amz-bedrock-kb-source-uri", "value": "s3://$bucket_name/video_game/neo_tokyo_games/" } }

2. Then, when calling the retrieve_and_generate method, include this filter in your request:

   response = bedrock_runtime.retrieve_and_generate( knowledgeBaseId='your-knowledge-base-id', input={ 'text': 'Your query here', 'retrievalConfiguration': { 'vectorSearchConfiguration': { 'filter': publisher_filter } } }, # other parameters as needed )

This approach will filter your retrieval to only include documents from the specified S3 prefix, which in this case is the "neo_tokyo_games" folder within your "video_game" directory.

Remember that the S3 prefix filtering is just one way to use metadata filtering. You can also define custom metadata for each document when ingesting data into the knowledge base, which allows for more flexible and specific filtering options based on your particular use case.

If you need to apply more complex filters or combine multiple conditions, you can use other operators like "andAll" or "orAll" to create more sophisticated filtering logic.

Sources
Amazon Bedrock Knowledge Bases now supports metadata filtering to improve retrieval accuracy | AWS Machine Learning Blog
Knowledge Bases for Amazon Bedrock now supports metadata filtering to improve retrieval accuracy | AWS Machine Learning Blog
RetrievalFilter - Amazon Bedrock

Hey I have a similar use case, I am using this start with filter to find specific user's folder in my s 3 bucket, however if the filter finds nothing, instead of returning en empty array its basically giving out other users folders too, can you help with this please?