Failure to filter string with date metadata on Bedrock with Opensearch Serverless
Context
I'm using Bedrock Knowledge Base with Opensearch Serverless without any index. Ingestion from S3 and retrieval are fully working, and filtering is working fine except for the issue related here.
Is this a bug or am I missing something here? Tried a few things but I think this should work from the get-go.
Expected Behavior
Since there isn't any "DATE" type, when requesting filtering by "date-like" metadata attribute with any valid operator, the filter works as usual and infer the attribute as a string.
Actual Behavior
Considering the metadata attribute with name like "mydatestring" and values with format "YYYY-MM-DD" like "2023-05-01", filtering fails both on API and console with error "failed to create query: Field 'mydatestring.keyword' not found. Rewrite first"
Additional Resources
Example metadata file:
{
"metadataAttributes" : {
"mydatestring" : "2023-10-03",
"mydatelist": ["2023-05-01"]
}
}
Example filter (API)
"retrievalConfiguration": {
"vectorSearchConfiguration": {
"filter": {
"equals": {
"key": "mydatestring",
"value": "2023-10-03"
}
}
}
}
Example Console Error
Consider that data_publicacao is similar to mydatestring here:
Supposition
- Possibly a bug related with Opensearch not creating keyword entry: https://forum.opensearch.org/t/document-fields-do-not-always-get-keyword-entries/16474
- Newest
- Most votes
- Most comments
The issue you're experiencing with filtering string-based date metadata in Amazon Bedrock Knowledge Base using OpenSearch Serverless is a known problem. It appears to be related to how text fields are indexed in OpenSearch.
The error message "failed to create query: Field 'mydatestring.keyword' not found. Rewrite first" suggests that OpenSearch is trying to use a keyword subfield for filtering, which doesn't exist for your date string field.
This problem occurs because when Bedrock Knowledge Base creates the index in OpenSearch Serverless, it doesn't automatically create the necessary keyword subfields for text fields that are used for filtering.
To resolve this issue, you have a couple of options:
-
Create your own index in the OpenSearch Serverless collection before setting up the Bedrock Knowledge Base. When creating the index, make sure to include keyword subfields for all text fields you want to use for filtering. Here's an example of how you might define a field in your index mapping:
"mydatestring": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } -
Instead of using string fields for dates, consider using numeric fields to represent dates (e.g., as Unix timestamps). Numeric fields don't require keyword subfields and should work with filtering out of the box.
-
If possible, recreate your Knowledge Base and allow Bedrock to automatically add the filter mapping fields when ingesting your documents with metadata. Some users have reported success with this approach.
If you continue to face issues after trying these solutions, it may be worth reaching out to AWS support for further assistance, as this seems to be a recurring issue with Bedrock Knowledge Base and OpenSearch Serverless integration.
Sources
Bedrock knowledge base filtering using string fields returns exception - "failed to create query: Rewrite first" | AWS re:Post
Bedrock knowledge base filtering on string / text fields returns error - "failed to create query: Field 'name.keyword' not found Rewrite first" | AWS re:Post
answered a year ago
Relevant content
asked a year ago
