New crawler error: Number of s3 targets exceeding limit

Question

We are getting a new error for some of our crawlers.
```
The number of s3 paths passed as input has exceeded the limit. Your account is limited to 100 accounts per crawler.
```
This shows up on a number of accounts that we manage, and without warning. We have checked the following:
 - The docs do not mention any service quotas of this nature
 - There is nothing on the quota management dashboard relating to this quota.
 
Any ideas of what this could be caused by?

Answer

Hello,

Internally we have recently added a new limit on the number of S3 Targets to a maximum of 100 per crawler by default. We added this limit to the number of targets to bound the request body size and ensure the best performance of our APIs and this limit can be increased per request.

As this is a soft limit, a service quota increase request can be submitted to the Glue service team, the team will review it and make a decision. 
Please use the below URL's for the same:

https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html

Please feel free to reach out to us if you have any concerns.

Thank you!

Answer

This was a planned change by the service team to ensure the best performance of APIs. However, this change was not notified to customer in advance by the team. Please accept our apology for the inconvenience caused.

Regarding the Glue service Quota, it looks like it is not yet in AWS Quota console. However, if you faced this issue again then please create a  AWS support case so, engineer can escalate this issue to internal service team.

New crawler error: Number of s3 targets exceeding limit

Relevant content