Using the default Amazon Linux AMI for Amazon EMR long lived cluster

0

I am working on enabling automatic patching for one of the EMR cluster. I understand that with Amazon EMR release 6.6 and later, when you launch new Amazon EMR clusters with the default Amazon Linux (AL) AMI option, Amazon EMR automatically uses the latest Amazon Linux AMI. So when I update/create a cluster with EMR release 6.8, it will use the latest Amazon Linux AMI at the time of boot but what is the solution if we have long lived cluster.

AWS
asked 2 years ago472 views
1 Answer
0

Hello there,

Thank you for raising this question in re:Post.

I understand that you are asking about how to patch the instances running on a long living cluster to keep them up to date.

EMR does not have any provisions to apply patches on running instances, it can however run a bootstrap action where you can run an "yum update" or install required patch and as you may already know bootstrap is only executed at the instance launch time.Regardless, EMR does run "yum update" for Critical and Security patches at the time of instance launch, whether that is at the time of cluster creation or on the new instances during a cluster Resize.

While, technically if you had selected the option Update all installed packages on reboot (recommended) a restart of the instances would trigger 'yum update' for Critical and Security patches pushed by Amazon Linux, it is not recommended you do a restart as it introduces risk of applications failures and data loss.

You can use automated patching using SSM to push patches to EC2 instances in your account, provided it does not involve instance restarts in which case there is a risk of application failures and data loss.

In both cases the AMI selected at the time of cluster launch will not be updated to the latest available AMI, for that you would have to relaunch the EMR cluster. Hence, the recommendation would be to architect towards short lived EMR Clusters wherever feasible.

Hope this answers your query, please feel free to reply to us if you have follow up questions. Have a great day!

AWS
SUPPORT ENGINEER
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions