S3 Vector, big metadata error

• In S3 Vectors, by default, all metadata keys are considered filterable. This means that while performing a query on the index, you can specify filters to only return specific data. According to the S3 Vectors Limits docs, Filterable metadata per vector is up to 2 KB (2048 bytes). Additionally, total metadata per vector, filterable + non-filterable, is up to 40 KB per vector.

• To resolve the issue, create a new vector index, and include AMAZON_BEDROCK_TEXT and AMAZON_BEDROCK_METADATA as nonFilterableMetadataKeys. This way, the you can store up to 40kb of metadata per vector.

sample:

metadataConfiguration={
        'nonFilterableMetadataKeys': [
            'AMAZON_BEDROCK_TEXT',                  <<<< This shows how keys are based as nonFilterable
            'AMAZON_BEDROCK_METADATA'
        ]
    }

The error you're encountering is related to the size limit for filterable metadata in S3 Vectors. The error message "Filterable metadata must have at most 2048 bytes" indicates that the metadata you're trying to attach to your vector exceeds the maximum allowed size for filterable metadata.

By default, all metadata in S3 Vectors is filterable unless explicitly specified as non-filterable during vector index creation. Filterable metadata has a size limit of 2048 bytes per vector. Even though your metadata example {"source": "huggingface"} appears small, there might be an issue with how it's being processed or encoded.

Here are some potential solutions:

1. Make sure your metadata JSON is properly formatted. In your command, you're using escaped quotes (\") which might be causing issues. Try using single quotes around the entire JSON string and double quotes inside:

--metadata '{"source": "huggingface"}'

2. If you need to store larger metadata, consider using non-filterable metadata keys. These must be specified during vector index creation and can't be changed later. Non-filterable metadata keys are ideal for storing larger amounts of data that don't need to be searchable but can be returned with query results.

3. Check if there are any hidden characters or encoding issues in your metadata string that might be increasing its byte size.

4. Verify that the total size of your metadata (including keys and values) doesn't exceed the 2048-byte limit for filterable metadata.

Remember that S3 Vectors supports string, number, boolean, and list types of metadata, and the metadata is ideal for attributes that you want to filter on, such as categories, timestamps, or status values.
Sources
Metadata filtering - Amazon Simple Storage Service
Vectors - Amazon Simple Storage Service
Vector indexes - Amazon Simple Storage Service

I am using AWS bedrock and configured the s3 vector.

when I am trying to sync the document + metadata (startingestionjob & documentingestion) its not working. ValidationException: Invalid record for key 'someid': Filterable metadata must have at most 2048 bytes

my metadata.json file has this below content: {"metadataAttributes": {"source": "huggingface"}}

I have enabled cloudtrail and logging on bedrock but RCA can't be found out. even after that I am getting same issue.

Hello,
A thing to notice is that below syntax works:

s3vectors-embed put --vector-bucket-name "test-vector-bucket" --index-name "titan-vector-index-2" --model-id "amazon.titan-embed-text-v2:0"   
--text-value "Hello Mr Andersen !"  
--metadata '{"category": "test", "source": "huggingface", "tasks": "sentence similarity", "languages": "english", "modalities": "text"}'

while this one does not, and I have the same error (Filterable metadata must have at most 2048 bytes):

s3vectors-embed put --vector-bucket-name "test-vector-bucket" --index-name "titan-vector-index-2" --model-id "amazon.titan-embed-text-v2:0" 
--text "\test.json"  
--metadata '{"category": "test", "source": "huggingface", "tasks": "sentence similarity", "languages": "english", "modalities": "text"}'

The file test.json has 3.5 KB and is made of similar lines with this one:

{"query": "Does it work with t-mobile", "pos": [" I DONT RECOMMEND THIS PHONE!!!!"]}

When the size of the file is 85 B, the second syntax also works ! So, it seams that the file with TEXT is considered file with metadata ! Or the text is also added as metadata !

UPDATE:
The main cause for the error seams to be the index feature: each index key can't hold a text bigger than 2048 bytes

Hey all, check out https://github.com/awslabs/s3vectors-embed-cli/pull/9 - trying to squash this bug, this addresses the root cause. Thanks!