AWS Glue CoarseGrainedExecutorBackend ERRORS

Hello,

I understand that you are seeing the below error in the CloudWatch logs for multiple Glue jobs since 14th August 2023:

ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to : Driver x.x.x.x:x disassociated! Shutting down.

This error could appear due to multiple reasons, one of them being insufficient memory. Other causes of the Driver becoming unresponsive are:

1. Making a .collect() or .show() command which makes all executors send data to the driver, and all data is loaded on the driver. or
2. You are reading a large number of small files and the driver is not able to "track" all the files.

You can refer to the following external article for workarounds to how to mitigate this error if they fit your use case. [1]

However, without inspecting the logs for this run id, we won’t be able to provide a conclusive root cause due to which this error is occurring. In order for us to troubleshoot further by checking the glue job details using the backend tools, please feel free to open a support case with AWS using the following link [2] with the sanitized script, the job run id, Spark UI logs for the job run and any additional information if required and we would be happy to help.

Thank you!

References:

1. https://indatawetrust.blog/2017/09/11/spark-error-coarsegrainedexecutorbackend-driver-disassociated-shutting-down/
2. https://support.console.aws.amazon.com/support/home#/case/create