S3 Performance Improvement with Transit Gateway Peered architecture

I strongly suggest you replace the Interface endpoint with Gateway endpoint because it eliminates costs of Interface endpoints for transfers to/from a VPC and S3. Interface endpoints have 2 associated costs: 1/ hourly attachment charges plus 2/ data transfer costs. There is no cost for Gateway endpoints.

When you create a Gateway endpoint, a route is added to the route-table in the VPC routing connections but avoiding egress via NAT GW, avoiding those data processing charges.

As per the Internetwork traffic privacy section of the S3 User Guide: You can use two types of VPC endpoints to access Amazon S3: gateway endpoints and interface endpoints (by using AWS PrivateLink). A gateway endpoint is a gateway that you specify in your route table to access Amazon S3 from your VPC over the AWS network. Interface endpoints extend the functionality of gateway endpoints by using private IP addresses to route requests to Amazon S3 from within your VPC, on premises, or from a VPC in another AWS Region by using VPC peering or AWS Transit Gateway.

To take advantage of the lowest cost network path when using Enable private DNS only for inbound endpoints, a gateway endpoint must be present in your VPC. The presence of a gateway endpoint helps ensure that in-VPC traffic always routes over the AWS private network when the Enable private DNS only for inbound endpoints option is selected. You must maintain this gateway endpoint while you have the Enable private DNS only for inbound endpoints option selected. If you want to delete your gateway endpoint you must first clear Enable private DNS only for inbound endpoints.

Based on your scenario, there are several factors limiting your S3 throughput performance that aren't related to your Transit Gateway peering setup.

The key bottleneck appears to be related to S3 transfer flow rate limits rather than your network infrastructure. Each connection to Amazon S3 has a maximum transfer rate of 5Gbps, regardless of your instance bandwidth. To achieve higher throughput, you need to optimize your approach to S3 transfers specifically.

For high-throughput transfers between EC2 and S3, consider these optimization strategies:

1. Use byte-range requests: When downloading large objects, make concurrent requests for byte ranges at the granularity of 8-16 MB.

2. Optimize concurrent connections: As a general rule, make one concurrent request for each 85-90 MB/s of desired network throughput. For a 10 Gbps NIC, about 15 concurrent requests over separate connections is recommended. For faster NICs like your c5n.18xlarge (which has 100 Gbps capability), you would need to scale up concurrent requests accordingly.

3. Use optimized transfer tools: Consider using tools that support parallel transfers like the AWS CLI with AWS Common Runtime (CRT) enabled or third-party applications such as s5cmd.

4. Consider S3 Transfer Acceleration: If you're transferring data across regions, S3 Transfer Acceleration can help minimize latency by using CloudFront's globally distributed edge locations.

The fact that you're seeing similar performance (~2.2-2.3 GB/s) whether going through TGW peering or directly via interface endpoint suggests that the bottleneck is not in your network path but in how your application is interacting with S3. Your iperf3 test confirms that the network infrastructure can handle much higher throughput (98 Gbps).

When you increased to multiple instances, you saw aggregate throughput increase to 5 GB/s but with increased latency, which suggests you're approaching S3 service limits or experiencing resource contention.

To maximize S3 throughput, focus on optimizing your application's approach to S3 transfers rather than the network path.
Sources
Maximizing Amazon S3 transfer throughput to Amazon EC2 instances | AWS re:Post
Horizontal Scaling and Request Parallelization for High Throughput - Best Practices Design Patterns: Optimizing Amazon S3 Performance
Performance guidelines for Amazon S3 - Amazon Simple Storage Service