AWS Bedrock Crawler Job limit

The error you're encountering is related to the ingestion limits of the AWS Bedrock Web Crawler. Unfortunately, there isn't a direct way to bypass this job limit. However, there are a few approaches you can consider to address this issue:

1. Optimize your crawler configuration:

   • Review and update your exclusion filters to reduce the number of pages being crawled. This can help you stay within the ingestion limits.
   • Adjust the "maxPages" parameter in the WebCrawlerLimits configuration. This parameter sets the maximum number of web pages that can be crawled from your source URLs, up to 25,000 pages. If you haven't set this explicitly, try setting it to a lower value that falls within the limit.

2. Break down your crawling job:

   • Instead of crawling the entire website in one job, consider breaking it down into smaller, more manageable chunks. You could create multiple crawling jobs, each focusing on different sections of the website.

3. Use the WebCrawlerConfiguration options:

   • Utilize the "scope" setting in the WebCrawlerConfiguration to limit what is crawled. You can set it to "HOST_ONLY" to restrict crawling to a specific host, or "SUBDOMAINS" to include subdomains.
   • Make use of both "exclusionFilters" and "inclusionFilters" to more precisely control which URLs are crawled.

4. Consider alternative architectures:

   • If your use case allows, you might want to explore using AWS Step Functions to orchestrate multiple smaller crawling jobs. This approach can help you overcome some of the limitations of a single job while still achieving your overall goal.

Remember, these limits are in place to ensure fair usage and system stability. If these solutions don't meet your needs, you may want to reach out to AWS support to discuss your specific use case and see if they can offer any additional guidance or options.
Sources
WebCrawlerLimits - Amazon Bedrock
WebCrawlerConfiguration - Amazon Bedrock
Amazon Bedrock: Issues with hitting the 20 node limit | AWS re:Post