How do I update all Amazon EMR nodes after the bootstrap action?

Resolution

The bootstrap action occurs before Amazon EMR installs and configures applications, such as Apache Hadoop and Apache Spark. To change all cluster nodes after Amazon EMR installs and configures the applications, attach a bootstrap action script.

Update the following bash script with the changes that you want to apply to your nodes:

#!/bin/bash
set -xe 

EMR_SECONDARY_BA_SCRIPT=$(cat <<'EOF'
while true; do
NODEPROVISIONSTATE=` sed -n '/localInstance [{]/,/[}]/{
/nodeProvisionCheckinRecord [{]/,/[}]/ {
/status: / { p }
/[}]/a
}
/[}]/a
}' /emr/instance-controller/lib/info/job-flow-state.txt | awk ' { print $2 }'`

if [ "$NODEPROVISIONSTATE" == "SUCCESSFUL" ]; then
sleep 10;
echo "Running my post provision bootstrap"
NODETYPE=$(cat /mnt/var/lib/instance-controller/extraInstanceData.json | jq -r '.instanceRole' | awk '{print tolower($0)}')

if [  "$NODETYPE" == "master" ]
then
echo "In Master"
######ENTER YOUR CODE THAT SHOULD BE EXECUTED ON PRIMARY NODE######


elif [ "$NODETYPE" == "core"  ]
then
echo "In Core"
######ENTER YOUR CODE THAT SHOULD BE EXECUTED ON CORE NODE######


elif [ "$NODETYPE" == "task"  ]
then
echo "In Task"
######ENTER YOUR CODE THAT SHOULD BE EXECUTED ON TASK NODE######

fi

exit;
fi
sleep 10;
done
EOF
)

echo "${EMR_SECONDARY_BA_SCRIPT}" | tee -a /tmp/emr-secondary-ba.sh
chmod u+x /tmp/emr-secondary-ba.sh
/tmp/emr-secondary-ba.sh &> /tmp/emr-secondary-ba.log &
exit 0

Note: By default, the preceding script uses the hadoop user. If your changes require permissions, then update the user to sudo.

After you update the bash script, attach the script to your cluster as a bootstrap action script.

Related information

Understanding the cluster lifecycle

Create bootstrap actions to install additional software with an Amazon EMR cluster