Error while syncing knowledge base with Claude 3 Haiku v1 parsing strategy

The error you're encountering is related to a specific file size limit for the Claude 3 Haiku v1 parsing strategy in Amazon Bedrock Knowledge Base. This limit is not explicitly listed in the general service quotas, which is why you couldn't find it there.

The error message indicates that there's a maximum total file size limit of 104,857,600 bytes (which is equivalent to 100 MB) for the given parsing strategy. This limit applies to the combined size of all files you're trying to sync from your S3 bucket to the knowledge base using the Claude 3 Haiku v1 parsing strategy.

To resolve this issue, you have a few options:

1. Reduce the number of files in your S3 bucket or the total size of the files you're trying to sync. Ensure that the combined size of all files is less than 100 MB.

2. If possible, consider splitting larger files into smaller chunks that fall within this limit.

3. You may need to sync your data in batches if you have a large amount of data that exceeds this limit.

4. Check if there are other parsing strategies available that might have higher limits, though this depends on your specific use case and data requirements.

It's worth noting that different models or parsing strategies in Amazon Bedrock may have different limitations. The Claude 3 Haiku v1 parsing strategy seems to have this specific file size constraint to ensure optimal performance and processing of the data.

If this limit significantly impacts your use case, you may want to contact AWS support for further guidance or to inquire about potential workarounds or alternative solutions for handling larger datasets in your knowledge base.
Sources
Bedrock KnowledgeBase sync error, can't access S3 bucket | AWS re:Post
Bedrock knowledge base data source sync is not working | AWS re:Post

Hi there, you will find information on limits for knowledge base data here: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-ds.html#kb-ds-supported-doc-formats-limits

Each file can't exceed 50MB for text-based files, and 3.75 for image files.

You can find additional information on limits for Ingestion jobs under "Service quotas" on https://docs.aws.amazon.com/general/latest/gr/bedrock.html