How to limit the amount of emails relating to "ELB are failing with HTTP 5xx" messages?

Question

We have an SNS hooked up to the ELB alert for "Environment health has transitioned from Ok to Warning. 1.1 % of the requests to the ELB are failing with HTTP 5xx" which emails our team, however this is generating hundreds of emails, and when you generate hundreds of emails, you tend to ignore them.

Now obviously the best fix would be to better scale our infrastructure, but a sticking plaster is to only say we want emails when "1.1 % of the requests to the ELB are failing with HTTP 5xx" is actually some kind of defined percentage.  How can we change it so that it only triggers the SNS when the percentage is say over 10%? Or any other user defined percentage?

Answer

By default there is no such metrics on which you could construct an SNS Alert. However you can use the metrics maths to construct a maths function in following manner :

(HTTPCode_ELB_5XX_Count)/RequestCount * 100

Please refer to the following documents for more clarity on Maths expression and their usage [1,2].

Reference:
[1] Using Metric Math - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html 
[2] Using Metric Math - Adding a Math Expression to a CloudWatch Graph - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html#adding-metrics-expression-console

A visual example of how to create a metric math expression can be found at: https://aws.amazon.com/blogs/mt/amazon-cloudwatch-metric-math-simplifies-near-real-time-monitoring-of-your-amazon-efs-file-systems-and-more/

How to limit the amount of emails relating to "ELB are failing with HTTP 5xx" messages?

Relevanter Inhalt