By using AWS re:Post, you agree to the Terms of Use
AWS re:Postre:Post

How do I stream SES notifications across different Kinesis Data Streams accounts?

6 minute read
0

I want to set up Amazon Simple Email Service (Amazon SES) event notifications to send data to an Amazon Kinesis data stream in another account. How can I do this?

Short Description

To set up Amazon SES event notifications to an Amazon Kinesis data stream in another account, do the following:

1.    Start at the SES source. Update your configuration settings in SES to push to an Amazon Simple Notification Service (Amazon SNS) topic. The updated setting configures AWS Lambda to subscribe to the topic as a trigger.

2.    Subscribe to SNS.

3.    Set up cross-account access.

4.    Set up Lambda to send data to Amazon Kinesis Data Streams. Lambda then uses the code to grab the full event from the Amazon SNS topic. Lambda also assumes a role in another account to put the records into the data stream.

5.    Use the Amazon Kinesis Client Library (KCL) to process records in the stream.

Resolution

Start at the SES source

Create an SES environment and add your email account to the list. Verify the email account to send emails through the service to your own email address. Set up a configuration set for the emails in order to track all of the possible event types that can occur during testing. In the configuration set, set the destination to Simple Notification Service (SNS), using the default tracking domain.

Note: You can create the topic when creating your configuration set, or you can create it before creating the set.

For more information about configuration sets, see Using Amazon SES Configuration Sets.

Subscribe to SNS

To test the notification topic created, you can temporarily subscribe your own email address in the SNS topic. This ensures that you receive notifications of the messages that come through from SES. Note that you'll need to reference the subscription list when configuring Lambda.

Important: When testing, any of the openor clickevents can be tracked only in emails that are in HTML format. This is because AWS embeds an invisible GIF image, which contains a tracking token. This tracking token tracks any notification emails that are opened, submitting feedback for the event back to the topic.

After SES and SNS are integrated, you can verify whether the integration was successful by using the SES console. Choose Email Addresseson the left side of the SES console. Then, choose the send a Test Emailbutton and select rawas the email format.

Note: If you are using the formatted option, the email will not be sent in HTML. As a result, feedback for the openand clickemail events are also not sent.

You can use the following template in the console:

Subject: Amazon SES Raw Email Test
MIME-Version: 1.0
Content-Type: text/html
X-SES-CONFIGURATION-SET: SESxLMxKIN
<!DOCTYPE html>
<html>
<body>
<h1>This text should be large, because it is formatted as a header in HTML.</h1>
<p>Here is a formatted link: <a href="https://docs.aws.amazon.com/ses/latest/DeveloperGuide/Welcome.html">Amazon Simple Email Service Developer Guide</a>.</p>
</body>
</html>

In this example template, note that the configuration set name is included in the headers. The configuration set name is included to use the configuration for the chosen events. This configuration set sends you an email and notifications for all of the actions that are taken with the email in your inbox. For additional reading on SNS, see Setting Up Amazon SNS Notifications.

Set up cross-account access

Create the setup to send the messages over to another account. This requires roles and policies to allow you to assume roles in another account.

**Note:**You must have a role to assume in the destination account. The execution role for Lambda and the role that is going to be assumed must also be created. The role that is going to be assumed has the permissions to interact with the services needed.

Here's an example template for the account with SES and Lambda, which you'll need to attach to the Lambda execution role:

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::222222222222:role/role-on-source-account"
    }
}

Here's an example template for the account with the data stream, which should be attached to the new role that Lambda assumes:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::111111111111:role/my-lambda-executionrole-source-account"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

This template defines the trust relationship of a Lambda execution role. The Lambda execution role assumes the role that will deliver data to the Kinesis data stream.

**Note:**The role assumed by Lambda must have permissions to put records into a Kinesis data stream.

Set up Lambda to send data to Kinesis Data Streams

You can set up Lambda to send data to Data Streams by using a template like this:

from __future__ import print_function
import base64 #Kinesis only accepts base64 encoded payloads. 
import boto3  #Allow programmatic access to AWS services.
import json
print('Loading function')
def lambda_handler(event, context):
    # Setting up the assumeRole for cross-account
    sts_connection = boto3.client('sts')
    acct_b = sts_connection.assume_role(
        RoleArn="arn:aws:iam::[ADD_ACCOUNT_ID]:role/AssumeLambdaCrossAccount", #TODO: Add relevant role ARN
        RoleSessionName="cross_acct_lambda"
    )
    print(acct_b)
    
    ACCESS_KEY = acct_b['Credentials']['AccessKeyId']
    SECRET_KEY = acct_b['Credentials']['SecretAccessKey']
    SESSION_TOKEN = acct_b['Credentials']['SessionToken']
    
    #Gets the SNS message that was posted (cliks, opens, sends, deliveries etc.)
    message = event['Records'][0]['Sns']['Message'] #Fetching the message in the JSON structure received.
    print("From SNS: " + message) #Checking the message received from SNS.
    
    #TODO:Generate a random partitionKey for kinesis. 
    #If a random partitionKey is not used in a multi shard stream it will lead to performance problems
    #PartitionKey = RandomKey
    
    print("Add data to cross-account stream")
    kinesis_client = boto3.client('kinesis',region_name="us-east-1",aws_access_key_id=ACCESS_KEY,aws_secret_access_key=SECRET_KEY,aws_session_token=SESSION_TOKEN) #Initiates the kinesis service with cross-account credentials.. 
    kinesis_client.put_record(StreamName="CrossAccountTest",Data=json.dumps(message),PartitionKey="partitionKey")

In this Python code example, replace RoleArnwith your stream name and Region to fit your environment. This function takes the SNS message received from SES, assuming the role from the destination account. The assumed role then puts messages into the destination stream.

To configure Lambda as a subscriber to the SNS topic, you can follow these steps:

1.    Sign in to the Amazon SNS console.

2.    On the navigation pane, choose Topics.

3.    On the Topicspage, choose a topic.

4.    In the Subscriptionssection, choose Create subscription.

5.    On the Create subscriptionpage, verify the chosen Topic ARN.

6.    Choose AWS Lambdaas your protocol type.

7.    Enter the Amazon Resource Name (ARN) of your function to establish the endpoint.

8.    Choose Create subscription.

For more information about using Lambda as a subscriber to SNS, see Using Amazon SNS for system-to-system messaging with an AWS Lambda function as a subscriber.

If a message is published to an SNS topic that is subscribed by Lambda, Lambda gets invoked with the payload of the published message. For more information about how to create a sample message history store, see Invoking AWS Lambda functions via Amazon SNS.

Note: If you don't want to set up cross-account access, the Credentialssections can be removed and written to a local Kinesis data stream. For more information about using Kinesis with Lambda and Python, see Kinesis in the AWS Boto3 documentation.

Use the KCL to process records

You can use the KCL to process records in the Kinesis data stream. After the data is in the Kinesis data stream, you can build custom applications and implement Kinesis Data Analytics for analysis and visualization.

Topics
AnalyticsMedia Services
Tags
Amazon Kinesis
Language
English
AWS OFFICIAL
AWS OFFICIALUpdated 3 years ago
No comments

Relevant content