AWS cloudwatch log alarm which steps should I be looking at ?

Hi, did you look into contributor insights? If you create a rule to detect communication errors per camera id, another rule to detect communication established per camera id, you can combine them using metric math (e.g. graph e2 - e1, where e2 is the id of the 2nd rule that detects communication established and e1 is the id of the first rule). This would allow you to fetch the ones where the diff is positive, graph with a 5 min period should be close to what you are looking for? See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContributorInsights-GraphReportData.html for an example of contributor insight setup. The difficulty I see in your use case is to find a syntax that allows Contributor insights to pick your value, as your log is not JSON structured, it feels to me easier to write rules when logs are structured, but depends on how familiar you are with contributor insights way of parsing logs

1. Create a CloudWatch Log Group: Ensure your camera logs are being sent to a CloudWatch Log Group.

2. Create Metric Filters: You'll need to create two metric filters for each camera:

   • One to detect when a camera goes offline
   • Another to detect when a camera comes back online

   The filter pattern for offline events would look something like: [date, time, ..., cameraid="testc00013.test0001ev", ..., message="Device communication error"]

   And for online events: [date, time, ..., cameraid="testc00013.test0001ev", ..., message="Device communication established"]

   You'll need to create these filters for each unique camera ID.

3. Create Custom Metrics: When creating the metric filters, assign each a custom metric name. For example:

   • CameraOffline_testc00013_test0001ev
   • CameraOnline_testc00013_test0001ev

4. Create CloudWatch Alarms: For each camera, create an alarm that triggers if the camera is offline for more than 5 minutes. You can do this by:

   • Using the "CameraOffline" metric as the alarm trigger
   • Setting the threshold to 1 (indicating an offline event occurred)
   • Setting the evaluation period to 5 minutes
   • Using the "CameraOnline" metric to reset the alarm

5. Set up Notifications: Configure Amazon SNS (Simple Notification Service) to send notifications when alarms are triggered.

To implement this for hundreds of cameras, you may want to consider automating the process:

1. Use CloudWatch Logs Insights to query and identify all unique camera IDs in your logs.

2. Create a script or use AWS CloudFormation to automatically generate the necessary metric filters, custom metrics, and alarms for each camera ID.

3. Use AWS Lambda in combination with CloudWatch Events to dynamically create new alarms when new camera IDs are detected in the logs.

This approach allows you to create individual alarms for each camera, monitoring their offline status and alerting you if they don't come back online within 5 minutes. The automation suggestions can help manage this at scale for hundreds of cameras.
Sources
Alarming on logs - Amazon CloudWatch
Use CloudWatch alarms to monitor CloudTrail events | AWS re:Post
AWS announces CloudWatch Logs Anomaly Detection and Pattern analysis

Hello Andy, I saw your comment on the re:Post Agent post. Could you please share with me why the solution proposed does not work for you?