Skip to content

TGW and Privatelink MTU

0

Hi,

according to https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-quotas.html and https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-limits-endpoints.html both TGW and Privatelink endpoints support a MTU of 8500 bytes.

Both claim to perform MSS clamping.

Now I would expect the MSS to be 8460 bytes (given the MTU of 8500). In reality I mostly observed 8365 though (when traversing TGW and a Privatelink endpoint).

I have raised this question to AWS support and already similarly in re:Post and the reply was "yeah there is probably some encapsulation happening, hence lower MSS than expected". Honestly this does not make sense to me. If AWS decides to publish the MTU of a service as 8500, then this has to be the usable MTU and not just some internal MTU that AWS is using to then perform multiple encaps and then provide a much lower (unknown) MTU to the customer.

Can anyone give some insights?

asked a year ago484 views
4 Answers
0

The difference between the expected MSS of 8460 bytes (based on MTU of 8500) and the observed MSS of 8365 bytes is likely due to additional protocol overhead and encapsulation that occurs within AWS networking services.

Transit Gateway supports an MTU of 8500 bytes for traffic between VPCs, AWS Direct Connect, Transit Gateway Connect, and peering attachments. Similarly, NAT gateways support traffic with a maximum MTU of 8500 bytes. Both services enforce Maximum Segment Size (MSS) clamping for all packets.

When traffic passes through these services, there may be additional encapsulation required for routing and security purposes that consumes some of the available packet size. This encapsulation overhead reduces the effective payload size available for your data, resulting in a lower MSS than the theoretical maximum.

It's important to note that while AWS publishes the MTU as 8500, this represents the maximum packet size that can traverse the service, including all headers and encapsulation. The usable payload space (reflected in the MSS) will necessarily be smaller to accommodate these headers.

For context, when migrating from VPC peering (which supports jumbo frames with MTU of 9001 in the same region) to Transit Gateway, you previously had to manually configure the MTU to less than 8500 on EC2 instances to prevent traffic from being silently dropped. Recent enhancements to Transit Gateway now support Path MTU Discovery (PMTUD), which helps manage these MTU differences automatically by sending appropriate ICMP messages when there's an MTU mismatch.

If you're experiencing specific performance issues related to the MSS value, you might want to consider adjusting your application to work optimally with the observed MSS rather than the theoretical maximum.
Sources
Amazon VPC Transit Gateways Quotas - Amazon VPC
Performance and metrics enhancements for AWS Transit Gateway and AWS Cloud WAN | Networking & Content Delivery
NAT gateway basics - Amazon Virtual Private Cloud

answered a year ago
0

When traffic passes through these services, there may be additional encapsulation required for routing and security purposes that consumes some of the available packet size. This encapsulation overhead reduces the effective payload size available for your data, resulting in a lower MSS than the theoretical maximum.

That is why the MTU is not 9001 but much lower (8500)! Still, it does not explain why the MTU of 8500 is advertised when the actual (IP) MTU seems to be 8405.

answered a year ago
0

I tested again:

  • TGW will do MSS clamping to 8460 as expected. Even with two TGWs in the path, it stays at 8460.
  • A private endpoint -> private link service -> NLB will do MSS clamping to 8365! According to the documentation it should do 8460 instead.
  • In a centralized inspection setup with a TGW + appliance mode, then an Inspecion VPC with GWLB endpoints -> GWLB > NVA appliance I see this behavior:
    • SYN arrives at FW NVA with MSS clamped to 8365 (like I tested, its probably the private endpoint of GWLB type doing that)
    • SYN packet goes back from FW with MSS still at 8365.
    • SYN packet arrives at destination with MSS now at 8341 (something clamped it down by 24 bytes again!?)
answered a year ago
0

Your observation is correct, it seems the GWLB endpoint is clamping MSS at a lower value (8365 bytes). We will work on getting this documented accurately.

AWS
EXPERT
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.