How do I resolve the ErrorPortAllocation error on my NAT gateway?

Short description

NAT gateways support up to 55,000 simultaneous connections to each destination. If this threshold is crossed, then new connections to the destination fail and the ErrorPortAllocation metric for the NAT gateway increases in Amazon CloudWatch.

To resolve this issue, find the source clients and destinations that are causing these errors. Then, follow the steps in the Resolve port allocations errors Resolution section.

Prerequisites:

• Make sure that the VPC flow log is activated at the VPC level or to the subnets where the NAT gateway is used.
• Make sure that the VPC flow log is configured to deliver to CloudWatch.

Resolution

Find source clients and their connection destinations

1.    Open the CloudWatch console.

2.    In the navigation pane, choose Insights.

3.    For Log group, choose the log group where the flow logs are recorded.

4.    Find where traffic is going:

filter (srcAddr like 'NAT gateway Private IP' and dstAddr not like 'xxx.xxx.')
| stats count(*) as numaccept by dstAddr
| sort numaccept desc
| limit 10

The results show the destination IP with the greatest number of responses during the time frame when you received the port allocation errors.

Note: Replace xxx.xxx. with the first two octets of your VPC CIDR. Also, replace NAT gateway Private IP with the private IP of your NAT gateway.

5.    Find the source clients that are sending traffic to the destination:

filter (dstAddr like 'Public IP from above filter' and srcAddr like 'xxx.xxx.') 
| stats sum(bytes) as bytesTransferred by srcAddr, dstAddr
| sort bytesTransferred desc
| limit 10

Note: Replace xxx.xxx. with the first two octets of your VPC CIDR. Also, replace Public IP from above filter with the destination IP that you received from the preceding query.

Resolve port allocation errors

• Create a NAT gateway in each Availability Zone, and then distribute your clients across Availability Zones. Route traffic to the internet using a NAT gateway in the same Availability Zone as your client to reduce cross Availability Zone data charges.
• If you notice an increase in the IdleTimeoutCount metric in CloudWatch, then configure your application or private instance to close idle connections. This lets the NAT gateway allocate the source port to new connections.
• Limit the number of connections that your clients can make to a single destination.
• For traffic going to an Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB public IP address in the same AWS Region, use a gateway VPC endpoint. There are no data processing or hourly charges for using gateway VPC endpoints.

---

Related information

Sample queries

Monitoring NAT gateways using Amazon CloudWatch