Receiving outdated CloudWatch Alarms, why?

There are several possible reasons why you might be receiving CloudWatch alarm notifications for events that occurred days ago:

1. Delayed Metric Data: CloudWatch alarms evaluate metrics based on data points available during the evaluation period. If your CloudTrail logs are being ingested with a delay, the metric filter might be processing older events, causing the alarm to trigger for historical data.

2. Metric Timing Issues: If metrics are published a few minutes or hours late, the alarm might be evaluating data that appears "historical" from your perspective. This can happen with log-based metrics where processing delays occur.

3. Alarm Configuration: Check your alarm configuration, particularly the period and evaluation periods. The way data gets aggregated for alarm evaluation might differ from how you're visualizing it in the CloudWatch console.

4. Insufficient Data Handling: Verify how your alarm is configured to handle missing data. This setting can affect when and how alarms trigger.

To troubleshoot:

• Check the alarm history to see the exact times when status changes occurred
• Examine the alarm configuration details, especially period settings and evaluation criteria
• Verify that the metric filter is correctly defined and processing logs as expected
• Consider adjusting your alarm to use "M out of N" evaluation (requiring multiple data points to exceed threshold) which can help with sporadic data issues
• Review the CloudWatch agent logs if applicable to identify any processing delays

If after checking these aspects you still experience issues, you might want to modify your alarm configuration to be less sensitive to delayed data points or contact AWS Support for further assistance with your specific setup.
Sources
Cloudwatch alarms are not triggering | AWS re:Post
Cloudwatch Alarms Bug | AWS re:Post

This might be related - can you check the email message source and find out when it was actually sent. It could be the messages were in your email firewall and later got approved and sent. To verify, go to SNS and manually publish a message and see if you get it instantly in your inbox.