How can I collect custom metrics from Amazon EMR cluster instances and then monitor them in CloudWatch?

Resolution

Note: If you use Amazon EMR version 7.0.0 or later, then install CloudWatch Agent as an application. For more information, see Amazon CloudWatch agent.

To configure and collect metrics from an Amazon EMR cluster, install the CloudWatch agent on Amazon Elastic Compute Cloud (Amazon EC2). After you install the CloudWatch agent, use the metrics collected in addition to the default published metrics for Amazon EC2 instances.

Store and view the metrics that you collect with the CloudWatch agent in CloudWatch similar to any other CloudWatch metrics. The default namespace for metrics collected by the CloudWatch agent is CWAgent. However, when you configure the agent, you can specify a different namespace.

To configure the CloudWatch agent and publish custom metrics from the Amazon EMR cluster nodes, complete the following steps:

1. Create an example CloudWatch agent configuration file (config.json) with the following example configuration. In the following example, four CPU metrics, two disk metrics, and one memory metric collect the metrics for a Linux server. The agent is set up to receive these metrics from a collectd client:

   {
     "agent": {
       "metrics_collection_interval": 60,
       "run_as_user": "root"
     },
     "metrics": {
       "aggregation_dimensions": [
         [
           "InstanceId"
         ]
       ],
       "append_dimensions": {
         "ImageId": "${aws:ImageId}",
         "InstanceId": "${aws:InstanceId}",
         "InstanceType": "${aws:InstanceType}"
       },
       "metrics_collected": {
         "collectd": {
           "metrics_aggregation_interval": 60
         },
         "cpu": {
           "measurement": [
             "cpu_usage_idle",
             "cpu_usage_iowait",
             "cpu_usage_user",
             "cpu_usage_system"
           ],
           "metrics_collection_interval": 60,
           "resources": [
             "*"
           ],
           "totalcpu": false
         },
         "disk": {
           "measurement": [
             "used_percent",
             "inodes_free"
           ],
           "metrics_collection_interval": 60,
           "resources": [
             "*"
           ]
         },
         "mem": {
           "measurement": [
             "mem_used_percent"
           ],
           "metrics_collection_interval": 60
         },
         "statsd": {
           "metrics_aggregation_interval": 60,
           "metrics_collection_interval": 10,
           "service_address": ":8125"
         }
       }
     }
   }

2. Copy the following bootstrap script to your Amazon Simple Storage Service (Amazon S3) location:

   #!/bin/bash
   
   sudo yum install amazon-cloudwatch-agent -y
   sudo amazon-linux-extras install collectd -y
   
   aws s3 cp <s3 path for config.json> /home/hadoop/config.json
   
   sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:///home/hadoop/config.json

   The script first installs the CloudWatch agent collectd client. Then, it copies the config.json file from Amazon S3 to the local path /home/hadoop. Finally, the script runs the CloudWatch agent with the config.json file.

3. Add the bootstrap action to the Amazon EMR cluster.

After you import the collectd metrics into CloudWatch, you can view these metrics as time series graphs. You can also set alarms so that you're notified when the metrics exceed a threshold that you specify.

To view the collectd metrics in the CloudWatch console, choose the namespace for the metrics that are collected by the agent. By default, this namespace is called CWagent. For more information, see Viewing collectd metrics imported by the CloudWatch agent.