EMR main instance is not reachable. Please check your instance status.

Question

Intermittently, the EMR parent node gets unreachable with the below error, which impacts the scaling of worker/task nodes and ultimately leads to batch job failures after few days (even the EMR CW monitoring metrics stop getting logged during this time).

"WaitingResize for group (ig-********) is pending because parent instance is not reachable. Please check your parent instance status."

We tried running the below command on parent node, which solves the problem temporarily, though again gets into the same situation after couple of days.

"/etc/alternatives/jre/bin/java -Xmx1024m -XX:MinHeapFreeRatio=10 -server -cp /usr/share/aws/emr/instance-controller/lib/*:/home/hadoop/conf -Dlog4j.defltInitOverride aws157.instancecontroller.Main &"

What would be the root cause?

Mr_Mishra · Answer

Hello
you might be aware that master node is responsible for pushing the cloudwatch metrics. So, if instance controller process is having issues, then the cloudwatch metrics and the error `instance is not reachable` can happen.

You can consider to check them on EMR master node as below

```
$ sudo systemctl status instance-controller.service
$ less /emr/instance-controller/log/instance-controller.log
```

EMR main instance is not reachable. Please check your instance status.

Ajoutez votre réponse

Contenus pertinents