Skip to content

What happens after reaching ingestion job size quota?

0

I see that the maximum size of the ingestion job is 100 GB. What happens if I reach this limit? In particular, what happens if I have more than 100 GB of data in my s3 bucket waiting to be ingested? Will Bedrock just report an error and stop ingestion, or will it ingest maximum of 100 GB of data and leave the remaining files until the next ingestion?

1 Answer
-1

When you reach the ingestion job size quota of 100 GB for Amazon Bedrock, the service will not automatically process the remaining data beyond this limit. The behavior you're asking about is not explicitly stated in the provided sources for Bedrock. However, based on general AWS service behaviors, it's likely that Bedrock will report an error and stop the ingestion process once the 100 GB limit is reached.

To handle data volumes exceeding 100 GB, you would typically need to manage the ingestion process manually by splitting your data into multiple ingestion jobs, each staying within the 100 GB limit. This means you would need to organize your S3 bucket content or control the ingestion process to ensure each job doesn't exceed the quota.

It's important to note that Bedrock also has limits on concurrent ingestion jobs. There's a maximum of 5 concurrent ingestion jobs per account, 1 concurrent job per data source, and 1 concurrent job per knowledge base. These limits further emphasize the need for careful planning when ingesting large volumes of data.

If you find that these limits are constraining for your use case, you may want to consider requesting a quota increase. While some quotas in AWS services are adjustable, others are not. For Bedrock-specific quota increases, you would need to contact AWS support or your account manager to discuss your needs and potential solutions.
Sources
Amazon Bedrock endpoints and quotas - AWS General Reference

answered 2 years ago

  • Can someone confirm it?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.