High latency with S3 Multi-Region Access Point (MRAP)

Question

Hey, we have 2 buckets - one in us-east-2 and one in ap-northeast-1 - behind an MRAP. I'm trying to troubleshoot some intermittent high latency issues with a client in a large communist country in Asia that I'm not allowed to name here for some reason attempting to PUT/GET files to/from the endpoint.

I've looked at the access logs on both buckets and the requests are definitely going to the ap-northeast-1 bucket rather than the us-east-2 one. The logs say that all the problem requests are being completed in under 1s ("Total Time" as per the [docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html)) but the HAR file the client sent me tells me a different story (can be anything up to 70s).

In order to troubleshoot this I really need to understand the discrepancy between the timings in the logs and the HAR, but the documentation on what happens under the hood in an MRAP is all but nonexistent. All I can find is that it uses a Global Accelerator (GA), and in the GA docs it says ["Global Accelerator terminates TCP connections from clients at AWS edge locations and, almost concurrently, establishes a new TCP connection with your endpoints."](https://docs.aws.amazon.com/global-accelerator/latest/dg/introduction-how-it-works.html) So does this mean that Total Time is measuring the time taken to transfer to the bucket from the endpoint for PUTs (and vice-versa for GETs)?

I've tried to find some logs for the MRAP architecture but the [docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPointMonitoring.html) waffle on and on about how requests are logged but doesn't say how to actually do it... I think it's referring to the S3 access logs I already have.

You can usually [enable flow logs](https://docs.aws.amazon.com/global-accelerator/latest/dg/monitoring-global-accelerator.flow-logs.html) for a GA but no GAs show up in the console, presumably because it's just a hidden component of an MRAP.

Anyone have any clue how this works?

re:Post Agent · Answer

To address the high latency issues you're experiencing with your S3 Multi-Region Access Point (MRAP), it's important to understand how MRAP works and how to troubleshoot the discrepancies you're seeing between S3 access logs and client-side measurements.

MRAPs use AWS Global Accelerator to route requests dynamically to the nearest S3 bucket, considering factors like network congestion and the location of the requesting application. This can result in improved performance, especially for requests routed over the public internet.

The discrepancy between the S3 access logs and the client's HAR file is likely due to the way MRAP and Global Accelerator handle connections. The "Total Time" in S3 access logs indeed measures the time taken to transfer data between the AWS edge location and the S3 bucket, not the entire round trip from the client. This explains why the logs show completion times under 1 second while the client experiences much longer delays.

To get a more comprehensive view of the request flow:

1. Use AWS CloudTrail for logging MRAP requests. CloudTrail logs all requests made through a Multi-Region Access Point, including the hostname of the MRAP. To enable CloudTrail logging for MRAP operations, configure it in the US West (Oregon) Region, regardless of your location or the Regions supported by your MRAP.

2. Monitor Amazon CloudWatch metrics for your MRAP to get insights into its performance and usage patterns.

3. Consider using AWS X-Ray to trace requests as they travel through your application and AWS services, which can help identify bottlenecks in the request path.

To improve performance for your client in Asia:

1. Ensure that your MRAP is configured to route traffic efficiently. Check that the ap-northeast-1 bucket is set as active for that region.

2. Implement aggressive timeouts and retries in your client application. The AWS SDKs have configurable timeout and retry values that you can tune to your application's specific needs. This can help maintain consistent latency, as retried requests are likely to take a different path and succeed quickly.

3. If applicable, consider using byte-range fetches for large objects to reduce latency and improve overall performance.

4. Evaluate if adding another bucket in a region closer to your client's location could improve performance.

Remember that while MRAPs aim to optimize performance, factors like internet routing and local network conditions can still impact overall latency, especially for clients in regions with restricted internet access.

**Sources**  
[Multi Region Access Points – Amazon S3 Features – Amazon Web Services](https://aws.amazon.com/s3/features/multi-region-access-points/)  
[Retry Requests for Latency-Sensitive Applications - Best Practices Design Patterns: Optimizing Amazon S3 Performance](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/retry-requests-for-latency-sensitive-applications.html)  
[Monitoring and logging requests made through a Multi-Region Access Point to underlying resources - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPointMonitoring.html)

High latency with S3 Multi-Region Access Point (MRAP)

Relevant content