Documents in queue with bedrock knowledge base custom datasource

When working with Amazon Bedrock's Knowledge Base feature and a custom data source, there is indeed a limitation on the number of concurrent document ingestion requests that can be processed at a given time. Specifically, the sum of concurrent IngestKnowledgeBaseDocuments and DeleteKnowledgeBaseDocuments requests cannot exceed 10 for an account.

This limit is a service-wide constraint and appears to be a hard limit set by Amazon Bedrock to ensure service stability and performance for all users. It's important to note that this limit applies to the actual API requests being processed, not to the number of documents that can be queued for ingestion.

To manage larger sets of documents for ingestion into a Knowledge Base with a custom data source, it's recommended to implement a queueing system and rate-limiting approach in your application. This would involve:

1. Creating a queue for your ingestion requests.
2. Setting up a pool of worker threads (maximum of 10) to process the queue.
3. Having each worker thread take a document from the queue, send the ingestion request, and wait for it to complete before processing the next document.
4. Implementing error handling and retries for failed requests.

By following this approach, you can effectively manage the ingestion of a large number of documents while staying within the service limits. The queue itself can hold as many documents as needed, but the actual processing will be limited to 10 concurrent requests.

It's also worth noting that for batch uploads of larger document sets, you might want to consider breaking your document set into smaller batches and processing them sequentially to optimize the ingestion process.

Remember to implement appropriate error handling and logging to track the progress of your ingestion process and manage any potential issues that may arise during the document ingestion.
Sources
Error: "sum of concurrent IngestKnowledgeBaseDocuments and DeleteKnowledgeBaseDocuments requests can’t exceed (10)" while ingesting documents to Bedrock Knowledge Base | AWS re:Post
Ingest documents directly into a knowledge base - Amazon Bedrock
New APIs in Amazon Bedrock to enhance RAG applications, now available | AWS News Blog