How do I push custom metrics to CloudWatch?

Short description

By default, AWS services push data points to CloudWatch. However, you might need to calibrate the performance of your resources based on metrics that aren't supported by AWS services. For these cases, use the CloudWatch agent or the API to push custom metrics to CloudWatch. Or, embed metrics within your CloudWatch Logs.

Important: Custom metrics are charged based on their storage and API use.

Resolution

Use the CloudWatch agent to push custom metrics

The CloudWatch agent collects system-level metrics and sends them to CloudWatch as custom metrics. Use the agent to push custom metrics from:

• Linux or Windows servers
• Amazon Elastic Compute Cloud (Amazon EC2) instances or on-premises servers

For Linux, see this list of supported metrics.

Example agent configuration file metric block for disk metrics (Linux):

"disk": 
    {
        "measurement": [
          
          "used_percent"
        ],
        "resources": [
          
          "*"
        ],
        "drop_device": 
        true
      }

For Windows, you can reference any counter mentioned in the Windows Performance Monitor in the agent configuration file.

Example agent configuration file metric block for Processor Counter (Windows):

"Processor": {
        "measurement": [
          {"name": "% Idle Time", "rename": "CPU_IDLE", "unit": "Percent"},
          "% Interrupt Time",
          "% User Time",
          "% Processor Time"
        ],
        "resources": [
          "*"
        ],
        "append_dimensions": {
          "d1": "win_foo",
          "d2": "win_bar"
        }
      }

To retrieve custom metrics from your applications or services, use StatsD and collectd protocols. Then, the metrics are pushed through the agent. StatsD is supported on both Linux and Windows servers. collectd is supported only on Linux servers.

To use these protocols, complete the following steps:

1. Install the CloudWatch agent.
2. Assign a role or credentials to the EC2 instance with CloudWatch permissions.
3. Create the CloudWatch agent configuration file.
4. Start the agent.

Use PutMetricData API to push custom metrics

If your use case doesn't support the use of the CloudWatch agent, then use the PutMetricData API to push custom metrics to CloudWatch.

For example, to push connections for a specific port as a custom metric, retrieve the values locally and transfer them in the API:

total_conn=$(netstat -an | grep <port> | wc -l) 
aws cloudwatch put-metric-data   --namespace "totalconn"   --metric-name <port> --dimensions Instance=<InstanceId> --value $  total_conn

Or, use AWS SDKs to use the PutMetricData API and send custom metrics to CloudWatch. Create a local script that runs periodically to send custom metrics. It's a best practice to use the API to push different metrics and multiple values for the following reasons:

• You can use up to 1,000 different metrics in a single API.
• If you use the Values and Counts method, then you can publish up to 150 values per metric with one PutMetricData request.
• You can use up to 30 dimensions per metric.
• Each PutMetricData request is limited to 1 MB for HTTP POST requests.

You can use a single API call to push multiple metrics, as shown in the following example:

aws cloudwatch put-metric-data --namespace "Usage Metrics" --metric-data file://metric.json

In this example, metric.json contains the following metrics:

[
  {
    "MetricName": "DiskMetric",
    "Value": <value_derived_by_some_calculation>,
    "Unit": "Count"
  },
  {
    "MetricName": "MemoryMetric",
    
    "Value": <value_derived_by_some_calculation>,
    "Unit": "Count"
  }
]

This method can reduce costs because custom metrics are charged based on the number of APIs and storage. The use of a single API call can also help to reduce throttling on the PutMetricData API.

Use CloudWatch Logs to generate custom metrics

To generate custom metrics based on your log data, use one of the following methods:

Embed Metrics in CloudWatch Logs

You can embed metric data into your log data. The embedded data is automatically pulled by CloudWatch to asynchronously generate custom metrics. You can also use this method to generate metric data from resources such as AWS Lambda or Amazon Elastic Container Service (Amazon ECS). An additional configuration isn't required to embed metric data.

The logs are in a JSON format and must have the root node _aws that contains CloudWatchMetrics, as shown in the following example:

{
  "_aws": {
    "Timestamp": 1725618677000,
    "CloudWatchMetrics": [
      {
        "Namespace": "custom-metric",
        "Dimensions": [["dimensionValue"]],
        "Metrics": [
          {
            "Name": "time",
            "Unit": "Milliseconds",
            "StorageResolution": 60
          }
        ]
      }
    ]
  },
  "dimensionValue": "value",
  "time": 100
}

For more information, see Embedding metrics within logs.

Use filters to create custom metrics from log events

To generate custom metrics based on the data present in your log events, use filters to create metrics from log events.