Cloudwatch Alarms Bug

Hi, elaborating a little bit on the answer shared above, please allow me to hint at a couple of checks:

• you mention that "the alarm gets set off by the Metric filter". Have you actually checked that the alarm status graph displays an alarm state in its timeline, and that the alarm history indicates the status change? If so, it’s a good thing and we’ll use it in my next check. If not, that would be a sign that the alarm is misconfigured in some way or not getting the metric values properly.
• if you have confirmed that the alarm status timeline and history display a status change, please navigate to the alarm history and open the details of the status change event and surrounding event: if there was a failure activating an event, it will be listed here along with an error message. This would help you understand what happened (for example wrong SNS topic name or the alarm privilege to post to that topic has been revoked) and thus how to fix it
• if on the contrary you see that the alarm status timeline and history didn’t detect any status change, the next steps would be to check if 1. the alarm sees the metric properly (simply check that the graph displays on the metrics details page - in your case this could happen if there is a mistake in the way the metric filter is defined, as hinted above), 2. the metric values are not "late" (if the metrics are published a few minutes late, the alarm will ignore them because the data will be too old) - for that you can experiment by asking the alarm to change state if 1 out of 4 or 5 datapoints fails - if you succeed making the alarm flip with such a change that’s a sign of possible delay - it is not likely to be your case obviously as we would detect ourselves if there was a situation where metric filters would be late ingesting data, and 3. the values are not aggregated the way you think, for example you’re using an AVG over 5 minutes in the alarm definition but when you display your metric you display an average over 1 minute periods, and thus the aggregation that you visualize is not the way the data gets aggregated for alarm evaluation. I hope those checks help you find the cause quickly and let you fix this easily.

I apologize, but I cannot look into your specific account or confirm any glitches in the AWS CloudWatch system. However, I can provide some guidance on troubleshooting CloudWatch alarms that don't seem to be invoking as expected.

First, it's important to verify that the expected logs are being generated and that the metric filter is correctly defined. Check that the log events include the expected values that match your metric filter pattern. You can test the pattern against example log events to ensure it's matching as intended.

Next, review the alarm configuration to make sure the correct math statistic value is set up. Sometimes, alarms may not trigger due to misconfigured metric parameters or insufficient data to determine the alarm state.

Also, consider the timing of data points. CloudWatch alarms evaluate metrics based on data points available during the evaluation period. If data points don't pass the threshold within the specified windows, the alarm remains in the OK state and won't trigger notifications.

If after checking these aspects you still believe there's an issue, I recommend contacting AWS Support for further assistance, as they can investigate your specific account and configuration.
Sources
Troubleshoot a CloudWatch alarm that doesn't invoke | AWS re:Post
Troubleshoot a CloudWatch alarm that monitors metrics | AWS re:Post
Using Amazon CloudWatch alarms - Amazon CloudWatch