How to fix duplicate transactional email issue caused by multiple EC2 instances?

0

A business guy's description: customers get multiple email notifications about the same transaction.

Developer's description: We have a problem with the load balancer and scheduled cron jobs in the Elastic Beanstalk ".ebextensions"-folder. When the load balancer scales in, there are multiple EC2 - instances and every instance executes cron jobs defined in the .ebextensions - folder of the project. That causes problems with the notification email delivery and our clients can receive multiple notifications -> sent email count is the same as the number of active instances. We've already tried scale in protection and "leader_only" = true definition for commands, but those are not working.

What should we do to fix this problem correctly?

2 Answers
2

The "cloud native" way of doing this is to use a queue.

It's not detailed in the question but my guess here is that there is a single submission to the application which is intended to cause a single email email notification. But the submission is (probably) stored in a database somewhere and when all of the instances run their cron jobs they all pick up the job out of the database and all send emails.

Instead: Have the submission of the job create a message in SQS and have the email notification system retrieve it from SQS. Even if there are multiple instances running the notification code only one of them will ever receive the SQS message - that's how the service is designed.

profile pictureAWS
EXPERT
answered 2 years ago
profile pictureAWS
EXPERT
Chris_G
reviewed 2 years ago
1

Hello,

Brettskis SQS Solution is probably the correct and more robust solution at scale. However, it could be a bit lite over-engineer for such an easy task.

Just create an endpoint in your service for "notifications" (the same function that your cron job triggered), then have a cron job in cloud watch event bridge trigger it using SNS, or go via a private lambda if it's a private endpoint. T

This way, only one request will be sent even if you have multiple EC2s, and it's easy to test and trigger add hoc "notifications". And you won't have to spend money on SQS pooling (keep in mind that SQS pooling blooks one thread in your machine).

Hope this helps!

//Carl

profile picture
answered 2 years ago
  • Given that SQS is free for the first million requests per month if this is a small-scale application then costs isn't going to be an issue. I run this exact solution for a personal website because it is inexpensive and it's really easy to engineer. Many AWS services are there to solve problems small and large. And SQS polling only blocks for the long polling period specified by the caller.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions