Skip to content

Bedrock Knowedge Base S3 OpenSearch Serverless Collection Metadata

0

I have setup Bedrock Knowledge Base with S3 and Opensearch Serverless as Vector DB in eu-central-1.

Uploaded PDF files in S3 bucket and added both, metadata (via properties) and the recommended (but only available in US regions as of 04.2024) additional metadata json file.

I added metadata attributes to my serverless collection index as well.

Whatever I do, the additional attributes are not being synced from the knowledge-base to the serverless collection index and thus cannot be found query the index.

Any idea ?

2 Answers
0

It seems you're experiencing an issue with metadata synchronization between your Amazon Bedrock Knowledge Base and the OpenSearch Serverless collection index. There are a few things to consider:

  1. Region Availability: You mentioned that you're using the eu-central-1 region. It's important to note that the metadata filtering feature for Amazon Bedrock Knowledge Bases is currently only available in US East (N. Virginia) and US West (Oregon) regions. This could be the primary reason why your metadata is not being synced or recognized.

  2. Metadata File Format: Ensure that your metadata files are correctly formatted and named. They should have the same name as the corresponding PDF file with a '.metadata.json' suffix. The content should follow the specified format, including the "metadataAttributes" structure.

  3. Metadata File Location: Make sure the metadata files are stored in the same folder or location as the source PDF files in your S3 bucket.

  4. Metadata File Size: Verify that your metadata files do not exceed the 10 KB limit.

  5. Sync Process: After adding or updating metadata, you need to perform a sync operation to ingest your data sources into the knowledge base. This process should update the vector store with the new metadata.

  6. Index Configuration: Double-check that your OpenSearch Serverless collection index is properly configured to handle the metadata fields you're trying to use.

Given that you're operating in eu-central-1, which is not currently listed as supporting the metadata filtering feature, this is likely the root cause of your issue. You may need to wait for the feature to become available in your region or consider using one of the supported US regions if possible.

If you continue to face issues after addressing these points, you may need to contact AWS support for further assistance, as there could be underlying service-specific limitations or configuration issues that are not immediately apparent.
Sources
Connect to Amazon S3 for your Amazon Bedrock knowledge base - Amazon Bedrock
Amazon Bedrock Knowledge Bases now supports metadata filtering to improve retrieval accuracy | AWS Machine Learning Blog

answered 2 years ago

0

Just think that I cannot be the only guy facing this issue outside the States. To sync metadata from e.g. S3 to and Opensearch collection and filter accordingly does not appear to be rocket-science.

So is there any documentation on a potential solution or should I rather fallback to the approach where I do all that manually ?

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.