Creating Custom Alarm

Hello.

If you use "--treat-missing-data notBreaching" as answered by AWS re:Post Agent, it will be considered normal even if there is no metrics data, so an alarm will not occur even if the container is stopped.
Therefore, I think it is necessary to change it to "--treat-missing-data breaching" and set it as an alarm if there is no metrics data.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data

• notBreaching – Missing data points are treated as "good" and within the threshold
• breaching – Missing data points are treated as "bad" and breaching the threshold
• ignore – The current alarm state is maintained
• missing – If all data points in the alarm evaluation range are missing, the alarm transitions to INSUFFICIENT_DATA.

Hello, you mention you have created an alarm and are looking for a way to include other dimensions.

Please note that: a standard metric alarm can only work if you specify all dimensions of your metric. You cannot add dimensions that your metric doesn’t have, and you cannot mention only a subset of the dimensions that your metric has. If your metric has only one dimension TaskDefinitionFamily, then you are right to try and set only one dimension. However, if your metric has 3 dimensions TaskDefinitionFamily, ClusterName & ServiceName, it is not possible for the alarm to work the way you defined it, as you provided a value for only one dimension, so the definition of your metric is ambiguous for the alarm.

If that reflects your case, there are 2 options:

1. either specify all the dimensions of your alarm. For that, in the aws cloudwatch put-metric-alarm command, replace --dimensions "Name=TaskDefinitionFamily,Value=f5gfins-sf-rocet1-api" with --dimensions "[{Name=TaskDefinitionFamily,Value=f5gfins-sf-rocet1-api},{Name=ClusterName,Value=pleaseUpdateTheValue},{Name=ServiceName,Value=pleaseUpdateTheValue}]"
2. or use a query-based alarm. A query allows you to match multiple metrics (so you can skip one dimension for example), aggregate them (for example taking the average or the max) and alarm on the aggregate. Alarming on the max of duration will allow you to know if there is at least one task that is taking too long to execute and breaches your duration threshold. One of the benefits of query-based alarming is that because you don’t need to specify all dimensions, it can match transient resources even if you don’t know ahead of time what the value of that dimension will be.

Also I’d like to share the following tips with you:

• when you create an alarm and your alarm stays in INSUFFICIENT_DATA state, please open the alarm in the console and check that it shows a metric graph on the alarm page. If no graph displays, it means the alarm is not properly defined (for example you didn’t specify all the dimensions of the metric)
• when you want to automate alarm creation from the command line or CloudFormation and are not sure of the syntax, you can create the alarm in the console, open the alarm page, and use the action menu to display the source of the alarm, it will show you the command line syntax and CloudFormation definition
• if you are not familiar with query based alarming, I suggest you open the "Metrics" menu in the console, click on the second tab (after browse, the next tab is queries), there is a query builder that will help you create a query. Run the query and it will help you check that you have the right definition and displays what you expect. When you are satisfied with the display, you can use the create alarm button on the query builder, even if you are displaying multiple metrics and not aggregating them, the alarm wizard will guide you through the following steps

Your approach to creating a custom alarm for monitoring ECS task execution time is on the right track, but there are a few considerations and potential improvements to address:

1. Alarm State: The "Insufficient data" state is expected when there are no active ECS tasks matching your metric dimensions. This is normal behavior for transient tasks that are not constantly running.

2. Dimensions: Your current configuration uses the TaskDefinitionFamily dimension, which is correct for tracking tasks across a specific task definition. However, you don't need to include ClusterName and ServiceName in the dimensions unless you want to narrow the scope of the alarm to a specific cluster or service.

3. Metric and Threshold: The ExecutionTime metric and the threshold of 1200 seconds (20 minutes) align with your requirement to trigger the alarm if a task runs for more than 20 minutes.

4. Period and Evaluation: Your current setup checks the average execution time over a 5-minute period (300 seconds) and requires two consecutive periods to exceed the threshold. This means the alarm will trigger if the average execution time exceeds 20 minutes for 10 minutes straight. You might want to adjust these values based on your specific needs.

5. Alarm Actions: You've correctly included an SNS topic ARN for notifications when the alarm triggers.

To improve the alarm's effectiveness for transient tasks, consider the following modifications:

1. Use a shorter period (e.g., 60 seconds) to capture shorter-lived tasks more accurately.
2. Adjust the evaluation periods to 1 if you want the alarm to trigger as soon as a single task exceeds the threshold.
3. Consider using the Maximum statistic instead of Average to capture the longest-running task in each period.

Here's an updated command incorporating these suggestions:

aws cloudwatch put-metric-alarm --alarm-name AWS_fm~ECS~f5gfins~F5G-TaskExecutionTime --alarm-description "Task Execution Time" --metric-name ExecutionTime --namespace AWS/ECS --statistic Maximum --period 60 --threshold 1200 --comparison-operator GreaterThanThreshold  --dimensions "Name=TaskDefinitionFamily,Value=f5gfins-sf-rocet1-api" --evaluation-periods 1 --alarm-actions "arn:aws:sns:us-east-1:accountnumber:topicname-topic" --region "us-east-1" --treat-missing-data notBreaching

The --treat-missing-data notBreaching option has been added to handle periods when no tasks are running, preventing false alarms during idle times.

Remember that this alarm will only capture data when tasks are actually running. For more comprehensive monitoring, you might want to consider additional alarms or metrics that track task starts, completions, and any failures.
Sources
Create alerts for EC2 instance events - AWS Prescriptive Guidance
Use PutMetricAlarm with an AWS SDK or CLI - AWS SDK Code Examples
AwsCloudWatchAlarmDetails - AWS Security Hub