How do I configure custom CloudWatch alarm notifications with EventBridge?

Short description

You can configure metric filters to match specific log events (for example, AWS CloudTrail events containing AccessDenied or UnauthorizedOperations errors). This allows you to derive metrics from Amazon CloudWatch Logs. You can then configure a CloudWatch alarm to monitor this metric.

By default, a CloudWatch alarm that uses Amazon Simple Notification Service (Amazon SNS) delivers a preformatted notification containing various alarm and metric details. Incorporating the log events that triggered the CloudWatch alarm into the notification gives you a better understanding of what caused the state change.

Leverage Amazon EventBridge to create a rule that triggers on a specific alarm in the ALARM state and invokes an AWS Lambda function. The Lambda function publishes a custom notification with the related log events.

Resolution

CloudWatch alarm doesn’t have a native mechanism to customize an alarm notification or message body.  The following section describes how you can build a custom alarm notification.

Prerequisites

• Make sure that you have an existing Amazon CloudWatch alarm that monitors a custom metric derived from a metric filter. For more information on how to configure a CloudWatch alarm, see Create a CloudWatch alarm based on a static threshold.
• For the EventBridge rule to trigger, create it in the same Region as the CloudWatch alarm.

Create an AWS Lambda function

You can configure a Lambda function to perform the following actions:

• Parse the CloudWatch alarm event details (for example, alarm name, description, timestamp, or the reason for state change). These details can be incorporated into the custom messaging.
• Analyze log data in the related CloudWatch log group using Log Insights. When the alarm changes state, the Lambda function issues:

  • StartQuery API call to run a query against the log group. For example, here's a query that summarizes AWS CloudTrail API calls that fail with the AccessDenied or UnauthorizedOperation error:

    filter (errorCode="AccessDenied" or errorCode="UnauthorizedOperation")
    | stats count(*) as Hits by errorCode, errorMessage, sourceIPAddress, userIdentity.arn
    | sort Hits desc

  • GetQueryResults to retrieve the corresponding results.

• Use SNS Publish API to deliver the custom message that contains the alarm details, along with the Log Insights query results.

Create an Amazon EventBridge rule

Amazon EventBridge deploys a rule that's configured to trigger when a CloudWatch alarm transitions to an ALARM state. When triggered, the EventBridge rule invokes a Lambda function to build and send a custom message. Follow these steps to build an EventBridge rule:

1. Open the Amazon EventBridge console.

2. In the navigation pane, choose Rules.

3. Choose Create rule.

4. Enter a Name and, optionally, a Description for the rule.

5. For Event bus, select AWS default event bus.

6. For Rule type, choose Rule with an event pattern.

7. Choose Next.

8. Under Event Pattern, create an event pattern similar to the one shown here:

   {
     "detail-type": [
       "CloudWatch Alarm State Change"
     ],
     "source": [
       "aws.cloudwatch"
     ],
     "detail": {
       "alarmName": [
         "CloudTrail_AccessDenied_UnauthorizedOperation"
       ],
       "state": {
         "value": [
           "ALARM"
         ]
       }
     }
   }

9. Choose Next.

10. Under Select target(s), select the Lambda function created in the earlier step. Then, choose Next.

11. Add any relevant tags to the rule, and then choose Next.

12. Review the rule configuration, and then choose Create rule.

Related information

How do I customize default Amazon SNS email messages?