CloudWatch Alarms based on specific log messages

Question

Hi,

I have been trying to figure out how I can set up CloudWatch alarms based on specific log messages. I have firewalls in HA pairs which export their logs to CloudWatch every minute or so. When the primary firewall goes down it fails over to the secondary one, generates a log message, and that message is pushed to CloudWatch as expected. The same thing happens when the primary firewall comes back up and everything reverts back to it. I want to have an alarm that will watch for these log messages and trigger an alarm I can send to SNS for notification purposes. I've tried a number of docs and tutorials, but none seem to work.

The format of the log message when the failover occurs is (as copied from CloudWatch):

```
{
    "serial": "112233445566",
    "timestamp": 1718729379,
    "msgid": 145,
    "categoryname": "High Availability",
    "priority": "Alert",
    "sourceport": 0,
    "destinationport": 0,
    "rxbytes": 0,
    "txbytes": 0,
    "firewallaction": "NA",
    "dpi": 0,
    "messagenotes": "Missed Peer heartbeat",
    "message": "Secondary firewall has transitioned to Active"
}
```

The message when it reverts back to the primary looks like:

```
{
    "serial": "AABBCCDDEEFF",
    "timestamp": 1718730087,
    "msgid": 144,
    "categoryname": "High Availability",
    "priority": "Alert",
    "sourceport": 0,
    "destinationport": 0,
    "rxbytes": 0,
    "txbytes": 0,
    "firewallaction": "NA",
    "dpi": 0,
    "messagenotes": "Preempt",
    "message": "Primary firewall has transitioned to Active"
}
```

I am mainly wanting to detect based on the "categoryname" and "message" fields.

Any help would be greatly appreciated.

Thanks,
smd75jr

Mahaboob Basha · Answer

Hello,
It seems you need a Cloud Watch alarms for logs in Cloud Watch. You cannot directly create alarms for logs but you have another way to create alarms.

Initially you have to create a metric for your log group, you can achieve from selecting specific log group you want and in Actions Drop down select  Create Metric Filter. You can select your required filter pattern and create a Metric for your Log Group.

Now you can easily create Cloud watch Alarms to Metrics and have SNS Notifications.

For more information Refer this document:-
[https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CreateMetricFilterProcedure.html]()

Leo K · Answer

With the built-in capabilities of CloudWatch Logs, I believe the best that can be accomplished is to create a metric filter: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringLogData.html

It will produce a CloudWatch metric, such as a simple 1 to indicate that the log entry represents a failover event and 0 to show that it doesn't. That numeric value can then be alerted on the same way as any regular numeric metric, like a CPU utilisation measurement.

What this won't allow doing is processing the individual log messages and taking action based on their contents, such as identifying the firewall that performed the failover and sending an email notification identifying the firewall. The ideal service for that wouldn't be CloudWatch Logs but EventBridge, which is an event service bus capable of filtering, routing, and duplicating events between services, but that won't help with logs delivered to CW Logs.

It's possible to construct selective log event processing based on CW Logs, but it would require combining multiple services. I think you'd have to start with subscription filters in CW Logs (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html) to send the log entries to another service. If the amount of log data is small, simply having it processed by a Lambda function to trigger the needed actions might suffice. If log data arrives at a high rate, the stream would probably first have to be sent to an Amazon Data Firehose stream, which would be able to buffer multiple messages and combine them into a batch before triggering the processing logic.

CloudWatch Alarms based on specific log messages

Add your answer

Relevant content