How do I find IP addresses of SMTP Clients behind a NAT gateway?

My SMTP Client is making TLS 1.0 / 1.1 requests to the Amazon Simple Email Service (SES) SMTP interface from behind a NAT Gateway. How do I find the instance behind the NAT gateway?

Short Description

Last year (June 2022), Amazon announced that TLS 1.2 will become the minimum TLS version used for AWS Service Public endpoints. Personal Health Dashboard (PHD) notifications were sent to customers if they use TLS 1.0 or 1.1 to send an SMTP email message.

SES provides two endpoints you can use to send emails. Users can invoke the standard AWS API, or an SMTP interface. SMTP is the protocol that third party products expect to use. These PHD notifications or messages from Amazon represent the email messages sent via the SMTP interface.

In order to continue to use SES uninterrupted, you need to upgrade the client (OS and/or third party software) to a version that supports TLS 1.2. In order to upgrade the client, you need to identify it. Many customers place their SMTP clients behind a NAT gateway, so when AWS reports the Client’s IP address, it reports the address of the NAT gateway, and not the client.

Resolution

SMTP clients communicate over ports 25, 587, 2587. They can also communicate over ports 465 or 2465 if they use the older TLS encryption protocol.

These ports are typically used to send SMTP emails, which means you can use VPC Flow Logs to identify the internal IP address of the clients.

In the next steps, we will create VPC Flow logs and send them to CloudWatch Logs and use CloudWtach Insights to search the SMTP events.

NOTE: Enabling VPC flow logs and sending them to CloudWatch Logs will incur extra charges to your account. Please review CloudWatch Pricing for more information.

Create a VPC Flow Log and send to CloudWatch Logs

If you are not using VPC Flow Logs, complete the following steps or see the documentation.

1. Open the Amazon VPC Console and click on VPCs
2. Select the VPC associated with the NAT gateway. If you have more than one VPC, you can determine the VPC associated with the NAT gateway by opening the Amazon VPC Gateway Console, click on NAT gateways. The VPC associated with your NAT gateway will be displayed.
3. Click on the Flow Logs tab
4. Click on Create Flow Log
5. For Filter select All,
6. Select a new CloudWatch Log Group — call it “VPC Logs” for easy identification
7. Create a new or select an existing role. If you create a new role, create a IAM and trust policy as described here.
8. Choose default format
9. Add Tags, if desired
10. Click Create Flow Logs

Use CloudWatch Insights to find the SMTP events

1. Open the Cloudwatch Console
2. Click on Logs Insights
3. Click Log Groups and select the log group that holds your VPC Logs
4. Select the time frame you wish to search

1. Write a query with a filter that selects only log entries that:
   • start from your VPC — you can do this with a regular expression
   • do not have your VPC as a destination — use the same regular expression with a not clause
   • use one of the SMTP Ports 25, 587, 2587
2. Add fields to display the:
   • source address (srcAddr)— this is the internal IP address of the client
   • destination address (dstAddr) — this is one of the SES addresses, or possibly another SMTP server
   • destination Port (dstPort) — the port used to communicate
   • the earliest time
   • the latest time

Example Query with the default 172.31.0.0/16 CIDR

filter dstPort in [25,587,2587] and srcAddr like /172\.31\..*/ and dstAddr not like  /172\.31\..*/
| stats min(@timestamp), max(@timestamp), count() by srcAddr, dstAddr, dstPort

3. Click Run Query.

4. The flow logs will show results starting from when you created it or from the time interval you chose. Check back periodically in case you have SMTP jobs that run periodically.

As noted above, sending VPC Flow Logs to CloudWatch Logs will incur additional charges. You can turn them off once you are done upgrading your SMTP clients and have confirmed resolution.

If desired, you can send email using a VPC endpoint. VPC endpoint will keep your traffic within the AWS network, so it will be faster, more secure, and in most cases, cheaper. The VPC endpoint will also provide the private IP address of the SMTP client. When you add the address, you do not need to make any other changes.

Conclusion

Once you have identified the source of your TLS 1.0 / 1.1 traffic, the next step is to upgrade the client. You can find more information about how to achieve this in our blog post.

For additional assistance, please open a support case or contact your Technical Account Manager if you have Enterprise Support.

Article co-authors:

• Andy Brand