High cost in NatGateway bytes

0

Hello Eveyone,

We found in Cost Explorer a large amount of data recorded in the metric USW2-NatGateway-Bytes in a single day. About 10TB. This is absolutely unusual and I would like to find the source and destination of all this traffic. So I created a CloudWatch dashboard of all my EC2 instances and configured it with various parameters in hopes of finding something that would show the source of the problem.

So far no parameter used has shed light on the issue and I still haven't figured out what happened.

I would like your help to try to identify where all this traffic came from.

Thank you!

2 Answers
2

Hi There

In addition to Tushar's answer, here is a Premium Support article that will walk you through determining the top talkers through your nat gateway after you enable VPC Flow Logs

https://aws.amazon.com/premiumsupport/knowledge-center/vpc-find-traffic-sources-nat-gateway/

If you did not have VPC flow logs turned on when the incident occurred, you might be able to find something by examining you EC2 instance CloudWatch metrics, particularly NetworkOut and NetworkIn for that particular day, but this may not be accurate because it will show ALL traffic in/out of the instance including to other instances and other AWS services.

Also take a look at this article after you determine the source for some suggestions on how to remediate and monitor in the future

https://aws.amazon.com/premiumsupport/knowledge-center/vpc-reduce-nat-gateway-transfer-costs/

profile pictureAWS
EXPERT
Matt-B
answered a year ago
  • Without any idea of what is going on in the VPC, my first guess would be that extra traffic is S3 uploads and/or downloads and there is no S3 gateway endpoint in VPC, but traffic goes from private subnets to NATGWs and then out to internet where S3 public endpoint is.

  • Thanks for your message, Matt-B. We didn't have the vpc log flow enabled so I'm trying my luck using CloudWatch data. The articles you sent were very handy, thank you!

1

You will need to enable VPC Flow Logs With additional Meta-Data such as below to find out the original Source and Destination IPs:

pkt-srcaddr : the packet-level IP address of the source. You typically use this field in conjunction with srcaddr to distinguish between the IP address of an intermediate layer through which traffic flows, such as a NAT gateway.

pkt-dstaddr : the packet-level destination IP address, similar to the previous one, but for destination IP addresses.

Reference: https://aws.amazon.com/blogs/aws/learn-from-your-vpc-flow-logs-with-additional-meta-data/

profile pictureAWS
EXPERT
answered a year ago
  • Thanks for your message, Tushar J. Unfortunately, VPC flow logs was not enabled until we find the issue. The article you sent was very useful, thank you!

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions