S3 Vector use-case: big files

The filterable part can’t be bigger than 2 KB, so in one of the links to questions and commands you are using I see you are passing the actual document text which is filterable and is more than 2kb so likely S3 will reject to save it. It’s common to split large documents into sections when working with AI search systems, so you can break you document in fine small chunks and for each chunk you save, add a simple metadata key (such as doc_id) containing the name of the original file.

When working with large documents (like 1MB files) in S3 Vectors, you should use non-filterable metadata for storing the larger content while using chunking strategies for effective semantic search.

Here's how to approach this:

1. Use non-filterable metadata for large content: While filterable metadata has a 2KB limit (or 40KB extended size), non-filterable metadata is designed specifically for storing larger amounts of contextual data. This is ideal for storing full document text, detailed descriptions, or other large content that doesn't need to be searchable but should be returned with query results.

2. Implement a chunking strategy: Divide your large document into smaller, meaningful chunks (paragraphs, sections, etc.). Each chunk would be represented as a separate vector in your index.

3. Store the chunks as vectors: For each chunk, create a vector embedding representing its semantic meaning, and store this in your S3 Vector index.

4. Use metadata effectively:

• Store the chunk's text content as non-filterable metadata
• Use filterable metadata for attributes you want to query on (document ID, section number, categories, etc.)

5. Retrieve and reconstruct: When performing semantic search, you'll get back the most relevant chunks. You can then retrieve the associated non-filterable metadata containing the actual text content.

This approach allows you to search across large documents by finding the most semantically relevant chunks, while still maintaining access to the full content through non-filterable metadata. The semantic search operates at the chunk level, finding the most relevant pieces of your document based on the vector similarity.

Remember that non-filterable metadata keys must be explicitly configured during vector index creation and cannot be changed to filterable later. You can configure up to 10 non-filterable metadata keys per vector index.
Sources
Metadata filtering - Amazon Simple Storage Service
S3 Vectors best practices - Amazon Simple Storage Service
Amazon OpenSearch Service vector database capabilities revisited | AWS Big Data Blog
Vector indexes - Amazon Simple Storage Service