"Resource Unavailable" in AWS Glue
I have around 80 AWS Glue jobs concurrently at peak in a given time window ( have already raised the default limit of 50 concurrent jobs to 150 ) but came to a scenario where the AWS Glue took 36 mins to respond back with "Resource Unavailable" error. These jobs run standard worker types and max capacity between 2-5 and the account limit on number of DPU(s) has already been bumped to 600.
Does any one have idea around the possible reasons for it and is it normally recommended to handle this error and resubmit the AWS Glue job ?
- 최신
- 최다 투표
- 가장 많은 댓글
For any AWS Glue jobs, connections and development endpoints that you use, check that your cluster has not run out of elastic network interfaces.
As per the online docs this could be due to not enough ENIs available. I have experienced this as well. Increasing number of automated retries in an AWS Glue job can also automatically take care of this error.
