How do I identify the CloudWatch log event that caused a CloudWatch alarm to trigger?

Resolution

To identify the CloudWatch log event that caused a CloudWatch alarm to trigger, use one of the following methods:

• Manually check the logs from the log group.
• Automate your notifications to include all log events.

Manually check the logs from the log group

Complete the following steps:

1. Navigate to the log group that contains the log events related to your notification.
2. Select Search log group or Search all log streams to show all the log events in different log streams together.
3. Select the timeframe of the notification according to your alarm's configuration. Enter the filter pattern in the Filter events box, and then select Enter.

All the log events that match your filter pattern in the timeframe that you selected appear in the console.

Automate your notifications to include all log events

Note: The following steps use two Amazon Simple Notification Service (Amazon SNS) topics. Topic-1 triggers the AWS Lambda function from the alarm, and topic-2 sends the notification.

To automate your notifications to include all log events, use an AWS Lambda function. Complete the following steps:

1. Create a CloudWatch alarm that monitors your required metric from the metric filter. Set Amazon SNS topic-1 as the target.

2. Create Amazon SNS topic-1. Add the Lambda function as a subscription.

3. Create a Lambda function that has a logic flow that includes two Amazon SNS topics. Make sure that the role that runs the Lambda function has the necessary permissions for CloudWatch Logs and Amazon SNS access. Use the following Lambda code:
   Note: Replace example-region with the required AWS Region and example-sns-topic with the required Amazon SNS topic.

   import boto3
   import json
   from datetime import datetime, timedelta
   import re
   def lambda_handler(event, context):
      message=json.loads(event['Records'][0]['Sns']['Message'])
      client = boto3.client('logs')
      sns = boto3.client('sns')
      response = client.describe_metric_filters(
        metric_name=message['Trigger']['MetricName'],
        metric_namespace=message['Trigger']['Namespace'])
      #Extract oldest timestamp from 'NewStateReason' field from alarm event
      stt=message['NewStateReason']
      output=str(re.search("\[(.*?)\]", stt).group(0))
      var=output.split(",")
      result=var[-1]
      a=result.split(" ")
      b=(a[-2]+"T"+a[-1])
      c=b[1:len(b)-2]
      start_utc_time=datetime.strptime(c,"%d/%m/%yT%H:%M:%S")
      st=((start_utc_time - datetime(1970, 1, 1)) // timedelta(milliseconds=1))
      end_utc_time = datetime.strptime(message['StateChangeTime'],"%Y-%m-%dT%H:%M:%S.%f+0000")
      et=((end_utc_time - datetime(1970, 1, 1)) // timedelta(milliseconds=1))
      fle = client.filter_log_events(
        logGroupName=response['metricFilters'][0]['logGroupName'],
        filterPattern=response['metricFilters'][0]['filterPattern'],
        endTime=et,
        startTime=(st-60000))
      sns_response = sns.publish(
        TopicArn='arn:aws:sns:example-region:123456789012:example-sns-topic',
        Subject='Details for Alarm - ' + message['AlarmName'],
        Message='Alarm Name: ' + str(message['AlarmName']) +'   '+ 'Account ID: ' +             str(message['AWSAccountId']) +'   '+'Region: ' + str(message['Region']) +'               '+str(fle))

4. Create the Amazon SNS topic-2 and include an email subscription. Then, update the TopicArn parameter in the sns.publish function of the Lambda code.