How to setup sns notification for cloudwatch alarm using CDK for AWS Glue job

Hi, after creating the numFailedJobsAlarm object, you can call addAlarmAction(...actions) or one of the other add<TargetStatus>action method(s) on it (see https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cloudwatch.Alarm.html#addwbralarmwbractionactions), something like that:

numFailedJobsAlarm.addAlarmAction(new SnsAction(notificationsTopic));

So wrapping up all the above things together:

import { SnsAction } from "@aws-cdk/aws-cloudwatch-actions";
import { Topic } from '@aws-cdk/aws-sns';
import  { EmailSubscription } from '@aws-cdk/aws-sns-subscriptions';

[…]

//cloudwatch alarm - that's what you shared already upfront
const numFailedJobsAlarm = new cloudwatch.Alarm(scope, '{DVRD replication} numFailedJobsAlarm', {
  metric: numFailedJobsMetric,
  threshold: 1,
  evaluationPeriods: 1,
});

// Adding an SNS action and an email registration - that's what we discussed
const notificationsTopic = new sns.Topic(this, 'Topic'); // not sure if you need to prefix with "sns." - I've seen code without it work
numFailedJobsAlarm.addAlarmAction(new SnsAction(notificationsTopic));
notificationsTopic.addSubscription(new EmailSubscription('yourself@yourdomain.com')); // Does not matter to do it before or after attaching the SNS action to the alarm

Hi,

yes 1 is problematic, if you create an alarm on a specific metric without specifying all dimension values, the alarm will not be able to identify the metric you want to alarm on, so it will not be able to retrieve any data and the alarm will not work. If you use default settings, the alarm will stay forever in the INSUFFICIENT_DATA state. You can see it if you open the alarm page in the console: you'll see that the graph remains empty and the alarm does not have any change or event in its history. If you are in that case, then it explains why your test is failing in point 2.

If you are using ELB and have some ELB metrics, you can check the values in the console or by calling the ListMetrics API. Whatever you want to do in the end, I suggest you adapt my example to test with a metric you know all the dimensions of. I used HTTPCode_ELB_4XX_Count from ELB just as an example, feel free to use a metric you are more familiar with.

That being said, you generally have two options:

1. if you want to alarm at the granular level (on a specific resource), you need to identify the full list of dimensions, and make sure you specify each dimension's value in the metric parameter of the alarm. It is really important that to work properly, your alarm must be created with EACH of the dimensions that you can see in the console or query with the ListMetrics API.
2. alternately, you can alarm at an aggregated level (across multiple resources) by using a query-based alarm instead. Instead of creating your alarm on a metric XYZ with its dimensions, you alarm on an expression and you give an SQL query as the expression. This allows you to aggregate multiple metrics based on the dimensions that you keep out of the WHERE clause of the query. Example below.

Here is an example to create an aggregated alarm. I am re-using the ELB example since you seem to have tried it. I create an alarm on the following expression: SELECT MAX(HTTPCode_ELB_4XX_Count) FROM SCHEMA("AWS/ApplicationELB", AvailabilityZone,LoadBalancer) WHERE AvailabilityZone = 'eu-west-1a' As you can see in the expression, I am only giving the AvailabilityZone, so the alarm will aggregate the data across all metrics with that AvailabilityZone, regardless of their LoadBalancer. Remember that this is only useful if you are OK to alarm on things like "is there at least one resource that breaches the threshold"? or "is the average of all resources in that group breaching a threshold"? It's efficient for spotting the existence of outliers, but it doesn't give you the possibility to take automated actions on every resource impacted.

The following block would create an aggregated alarm with an SQL expression - please only use if you are OK with an aggregated signal, otherwise if you need to alarm on a specific LoadBalancer, you need to look up its value and keep using a Metric with ALL dimensions specified.

// Expression with a Metric Insight (SQL) query
const numFailedJobsExpression = new cloudwatch.MathExpression({
  expression: "SELECT MAX(HTTPCode_ELB_4XX_Count) FROM SCHEMA(\"AWS/ApplicationELB\", AvailabilityZone,LoadBalancer) WHERE AvailabilityZone = 'eu-west-1a'",
  usingMetrics: {},
  period: cdk.Duration.minutes(1),
  label: '4xx Errors',
});

// An aggregated CloudWatch alarm using an SQL expression
const numFailedJobsAlarm = new cloudwatch.Alarm(this, 'Alarm on worst 4xxError breach', {
  metric: numFailedJobsExpression,
  threshold: 1,
  evaluationPeriods: 1,
});

Hi, registering the email addresses with the SNS topic is included as the last step of the example provided, you don't need to go through the UI for that step: notificationsTopic.addSubscription(new EmailSubscription('yourself@yourdomain.com')); Note that SNS will send a confirmation mail to the email address you want to register, and the recipient will have to click in the email to confirm the registration, but you don't need to go through the console for any of those steps.

Sorry I imported "Topic" directly so you should remove the "sns." prefix on the line that creates the topic: const notificationsTopic = new Topic(this, 'Topic');

Below a full working example (I am not using the eventbridge rule so skipped that part and used another metric instead, but the alarm creation part should be enough if you need guidance for that part alone).

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';

import { aws_cloudwatch as cloudwatch, aws_events as events } from 'aws-cdk-lib';
import { Topic } from "aws-cdk-lib/aws-sns";
import {SnsAction} from "aws-cdk-lib/aws-cloudwatch-actions";
import { EmailSubscription } from "aws-cdk-lib/aws-sns-subscriptions";

export class CdkpocStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    //cloudwatch metric
    const numFailedJobsMetric = new cloudwatch.Metric({
      namespace: 'AWS/ApplicationELB',
      metricName: 'HTTPCode_ELB_4XX_Count',
      statistic: 'Sum',
      dimensionsMap: {
        "AvailabilityZone": "hiding real value",
        "LoadBalancer": "hiding real value"
      },
    })

    //cloudwatch alarm - that's what you shared already upfront
    const numFailedJobsAlarm = new cloudwatch.Alarm(this, '{DVRD replication} numFailedJobsAlarm', {
      metric: numFailedJobsMetric,
      threshold: 1,
      evaluationPeriods: 1,
    });
    
    // Adding an SNS action and an email registration - that's what we discussed
    const notificationsTopic = new Topic(this, 'rePostDemoTopic'); // not sure if you need to prefix with "sns." - I've seen code without it work
    numFailedJobsAlarm.addAlarmAction(new SnsAction(notificationsTopic));
    notificationsTopic.addSubscription(new EmailSubscription('hiding real value')); // Does not matter to do it before or after attaching the SNS action to the alarm
  }
}

Once you deploy the stack, the email address you subscribed in the last line will receive a mail like this:

You have chosen to subscribe to the topic: 
"hiding real value"
To confirm this subscription, click or visit the link below (If this was in error no action is necessary): 
Confirm subscription

Ah sorry thought you had already figured that part out. So the email is not tied to the alarm itself, it is tied to the SNS topic.

When you configure the alarm, you attach an SNS action to it, to post the alarm notifications to a SNS topic. Then you need to register the email addresses with the SNS topic, which is a separate step from the alarm configuration. In the snippet above, notificationsTopic is an object you have created with something like notificationsTopic = new sns.Topic(this, 'Topic'); (see CDK doc for SNS here).

And to add an email registration to your topic you can do something like notificationsTopic.addSubscription(new EmailSubscription('yourself@yourdomain.com')); - there is a dedicated CDK doc for email subscriptions.