AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
Excessive collection size for Amazon Bedrock knowledge base respectively OpenSearch Serverless Vectorsearch
Dear fellow AWS users and experts,
I am seeking guidance on the typical storage sizes for knowledge bases in Amazon Bedrock. The storage size of our first collection seems excessive relative to the embedded and indexed content.
Our organization is in early-stage experimentation with Amazon Bedrock to build an LLM assistant for internal use with a custom knowledge base containing internal documents and specific information. These documents, primarily PDFs, total 550 MB and are stored in a dedicated S3 bucket.
Initially, I used the Quick Create a New Vector Store - Recommended option when configuring the knowledge base. However, I preferred a more descriptive name for the collection, so I subsequently created a collection on Amazon OpenSearch Serverless manually and configured the knowledge base to use it as the vector store. I mirrored all settings from the quick-created vector store, and the knowledge base functions as expected. However, the total size of the collection is disproportionately large compared to the embedded and indexed content, even with replication turned off.
Here are the specifics:
- Total size of internal documents (PDFs): 550 Megabytes
- Combined size of indices (Two separate ones for testing): 360 Megabytes
- Total size of the serverless collection: 560 Tebibytes
Can anyone help explain why this massive data increase occurs and suggest ways to improve storage efficiency? Is this input-to-output data size ratio of one million typical for Amazon Bedrock knowledge bases?
Thank you for your time and any assistance you can provide!
| Total collection size |
|---|
 |
| Index overview |
|---|
 |
| Details Index 1 |
|---|
 |
| Details Index 2 |
|---|
 |
- Tags
- Knowledge Bases for Amazon BedrockVector Engine for Amazon OpenSearch ServerlessAmazon OpenSearch ServerlessAmazon Bedrock
- Language
- English
asked 2 years ago736 views
- Newest
- Most votes
- Most comments
The storage size you're experiencing with your Amazon Bedrock knowledge base and OpenSearch Serverless vector store seems unusually large and is not typical for the amount of data you've described. A collection size of 560 Tebibytes for 550 MB of original documents is extremely disproportionate and suggests there may be an issue or misunderstanding in the configuration or reporting.
For Amazon Bedrock Knowledge Bases using OpenSearch Serverless as the vector store, the storage size should be more closely related to the size of your original documents, plus some overhead for the vector embeddings and index structures. Typically, you might expect the vector store to be a few times larger than the original data, but certainly not in the range of terabytes for megabytes of input.
Here are a few points to consider:
-
Verify the reported size: Double-check if the 560 Tebibytes figure is accurate or if there might be a unit error in the reporting. It's possible that the actual size is much smaller, and there's a display issue in the console.
-
Check collection settings: Review your OpenSearch Serverless collection settings, particularly around replication and sharding. Even though you mentioned replication is turned off, it's worth double-checking these settings as they can significantly impact storage usage.
-
Examine index configuration: Look at your index configurations, including the number of shards and any custom settings that might affect storage. The two separate indices you mentioned should not cause such a dramatic increase in size.
-
Monitor ingestion process: If you're continuously ingesting data or if there's an issue with the ingestion process, it could potentially lead to duplicate data or unnecessary storage usage. Ensure that your ingestion pipeline is working as expected.
-
Check for any unintended data: Verify that there isn't any unintended data being stored in your collection. This could include logs, metadata, or other information that you might not be aware of.
-
Contact AWS Support: If after checking these points you still see such a large discrepancy, it would be advisable to contact AWS Support. They can help investigate if there's a underlying issue with the service or your specific setup.
To improve storage efficiency, consider:
- Optimizing your document preprocessing to remove unnecessary content before ingestion.
- Using appropriate compression settings if available.
- Regularly monitoring and cleaning up any unnecessary data in your knowledge base.
In a typical scenario, the input-to-output data size ratio for Amazon Bedrock knowledge bases should be much smaller, usually in the range of 1:2 to 1:10, depending on the specific use case and configuration. A ratio of one million to one, as you're seeing, is definitely not typical and warrants further investigation.
Remember, efficient use of storage in vector databases is crucial for both cost management and performance optimization in RAG (Retrieval Augmented Generation) applications.
Sources
Create a knowledge base in Amazon Bedrock Knowledge Bases - Amazon Bedrock
Dive deep into vector data stores using Amazon Bedrock Knowledge Bases | AWS Machine Learning Blog
Hello! What was the solution you followed?
In my company we have a similar situation, we have the files in sharepoint and weigh about 50 GB and the size of the collection is around 650 GB. Currently we have an open case with aws support, but we have not had a concrete solution in 2 weeks. The solution that aws support told was to delete what we deployed and recreate it, but it took us 1 week to sync the knowledge base.
Does anyone know what is the specific documentation where this conversion is explained at the time of synchronization and the chunks are taken to opensearch? there should be some mathematical explanation for this.