MaxIngestionFileCountPerJob Error in Knowledge Base Syncing; No failed files

Based on your description, there are a few potential causes for the "MaxIngestionFileCountPerJob limit for given parsing strategy: 1000 reached" error you're encountering.

First, it's important to understand that the error message is indicating that the system is trying to process more than 1000 files in a single ingestion job, which exceeds the service limit.

Even though your lambda batching functionality is set to sync only up to 100 files at a time (which is well below the 1000 file limit), the issue might be related to how quickly you're triggering subsequent ingestion jobs. When a knowledge base sync job shows as "Completed" in the console, there might still be background processes finalizing the ingestion. If you immediately trigger another sync job, the system might still be processing files from the previous job, causing the cumulative count to exceed the limit.

Another possibility is related to incremental syncing behavior. When AWS Bedrock performs incremental syncing, it scans all files in the data source to determine which ones need to be processed. If there are files that weren't properly processed in previous syncs (even if they don't show as "failed"), the system might be attempting to reprocess them along with your new batch.

Empty files in your S3 bucket could also be a factor. As noted in a similar case, empty files might not be counted in the scan but could still affect the ingestion process.

Additionally, there's a limit of 10 concurrent ingestion and deletion requests for Bedrock Knowledge Base. If your automated resyncing is triggering multiple jobs in quick succession, you might be hitting this concurrent request limit, which could cause unexpected behavior.

To troubleshoot this issue:

1. Add a longer delay between batch syncs to ensure each job fully completes before starting the next one
2. Check for and remove any empty files in your S3 bucket
3. Consider implementing a more robust queuing system that waits for complete confirmation that a job is finished before starting the next one
4. Review your S3 bucket for any potential file issues that might be causing repeated processing attempts

If the issue persists, you might need to temporarily reduce your batch size further or implement a more gradual syncing approach to better understand at which point the limit is being triggered.
Sources
AWS Bedrock Knowledge Base Incremental Sync Issues | AWS re:Post
Error: "sum of concurrent IngestKnowledgeBaseDocuments and DeleteKnowledgeBaseDocuments requests can’t exceed (10)" while ingesting documents to Bedrock Knowledge Base | AWS re:Post
AWS Bedrock Knowledge Base not syncing when files are being over-written in S3 Bucket | AWS re:Post
Incremental Syncing of Metadata files | AWS re:Post