S3 TCP timestamps contrary to RFC 7323 / PAWS

Question

This started happening a day or so ago. In a TCP/HTTPS request to <bucket>.s3.us-west-2.amazonaws.com, the <SYN,ACK> from S3 has a TSval of 0. Then the TSval in the subsequent message from S3 has a huge jump. If that jump is greater than 1<<31, which it often is, then the RFC 7323 definition of the PAWS algorithm will treat the arriving segment as not acceptable.

This causes TLS connections to fail frequently for clients that have strict PAWS implementations.

According to RFC7323

TSval timestamps sent on <SYN> and <SYN,ACK> segments are used to initialize PAWS.

The PAWS mechanism also puts a strong monotonicity requirement on the sender's timestamp clock.

Which is not what the AWS S3 implementation is doing.

We've also observed that AWS S3 negotiates timestamps, and then seems to forget that it did so, stopping using timestamps mid-session.

11:56:41.557255 IP 192.168.1.63.50711 > s3-us-west-2-r-w.amazonaws.com.https: Flags [S], seq 12800648, win 64240, options [mss 1460,sackOK,TS val 497870 ecr 0,nop,wscale 1], length 0
11:56:41.597607 IP s3-us-west-2-r-w.amazonaws.com.https > 192.168.1.63.50711: Flags [S.], seq 2083609842, ack 12800649, win 64240, options [mss 1420,nop,wscale 8,nop,nop,sackOK,nop,nop,TS val 0 ecr 497870], length 0
11:56:41.597657 IP 192.168.1.63.50711 > s3-us-west-2-r-w.amazonaws.com.https: Flags [.], ack 1, win 64768, options [nop,nop,TS val 497910 ecr 0], length 0
11:56:41.601921 IP 192.168.1.63.50711 > s3-us-west-2-r-w.amazonaws.com.https: Flags [P.], seq 1:518, ack 1, win 64768, options [nop,nop,TS val 497914 ecr 0], length 517
11:56:41.634537 IP s3-us-west-2-r-w.amazonaws.com.https > 192.168.1.63.50711: Flags [.], ack 518, win 243, length 0
11:56:41.635033 IP s3-us-west-2-r-w.amazonaws.com.https > 192.168.1.63.50711: Flags [.], ack 518, win 243, length 0

Which is also contrary to RFC 7323

If a non-<RST> segment is received without a TSopt, a TCP SHOULD silently drop the segment.

Answer

This issue seems to be related to TCP Timestamps and how they are handled by AWS S3 servers. The behavior observed, where the TSval jumps significantly, could cause problems for clients with strict PAWS implementations. It appears that AWS S3 servers negotiate timestamps but may not consistently use them throughout the session, which is contrary to RFC 7323.

To address this issue, you can:

Contact AWS Support: Report the observed behavior to AWS Support for investigation and resolution. They can provide insights into any changes or updates in the AWS S3 implementation that may have caused this behavior.

Adjust Client Configuration: If possible, adjust the client configuration to relax the PAWS requirements or modify TCP Timestamp handling to accommodate the behavior observed from AWS S3 servers.

Consider Workarounds: Explore potential workarounds such as implementing retries or adjusting TCP settings on the client-side to mitigate the impact of the observed behavior until a resolution is provided by AWS.

By taking these steps, you can address the issue of TCP Timestamp handling discrepancies observed when communicating with AWS S3 servers.

S3 TCP timestamps contrary to RFC 7323 / PAWS

Relevant content