Intermittent issues accessing S3 assets through CloudFront

Question

Description:
Since Tuesday, September 10th, 2024, we have been experiencing intermittent issues accessing static assets stored in our Amazon S3 bucket (arn:aws:s3:::static.thearcweb.com) through CloudFront (arn:aws:cloudfront::372288094944:distribution/EKA442RZWO8H5). 
The issue occurs sporadically throughout the day and resolves itself after a few minutes. For example, our website (thearcweb.com) occasionally fails to load images like https://static.thearcweb.com/images/PROD/PROD-70f15f85-350f-4033-b561-01a9b7ea48bc.jpg, but the issue resolves automatically after a few minutes. We have confirmed the affected assets exist in the S3 bucket and have public read permissions. This issue is significantly impacting our production traffic when it occurs. We have not made any recent changes to our CloudFront distribution or S3 bucket that could have triggered this behavior. 
Invalidating the CloudFront cache does not resolve the issue.

Troubleshooting steps taken:

Verified the CloudFront distribution is pointing to the correct S3 origin (static.thearcweb.com)
    Confirmed the affected S3 assets exist and have public read access
    Ensured the CloudFront distribution is enabled and deployed
    Invalidated the CloudFront cache for affected assets (no impact)

We suspect there may be an underlying issue with CloudFront communicating with our S3 origin, but we have been unable to identify the root cause. Could this intermittent issue be related to a regional or global AWS service outage that occurred on September 10th or in the following days? 
We'd appreciate if you could confirm whether there were any reported issues with CloudFront or S3 in the ap-south-1 region that might have impacted our distribution. If not related to a broader service outage, we would greatly appreciate your help investigating this intermittent problem and providing recommendations to permanently resolve it. Please let us know if you need any additional information to assist with troubleshooting.

Tejas · Answer

@Aditi, did you ever resolve the above issue? If so, what was the resolution steps that were followed, as I'm currently experiencing similar behaviour across my platforms.

Leo K · Answer

Which cache policy is applied to the cache behaviour leading to the S3 origin in CloudFront? Importantly, is it set to cache content or to disable caching? If caching is disabled, every request will go to the S3 origin, both increasing your costs and potentially leading to intermittent issues if the request limits for S3's internal partitions in your bucket are exceeded.

Have you got S3 server access logs enabled for your bucket? The request rate throttling would show there as `SlowDown` errors (https://docs.aws.amazon.com/AmazonS3/latest/userguide/ErrorBestPractices.html#UsingErrorsSlowDown). Also, if the failures are caused by some other issue with S3, the logs would reveal specifics in most cases.

You can also estimate whether rate limiting is involved by looking at the metrics for your CloudFront distribution, and/or by enabling request metrics (https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) for your bucket. They will show separately from CloudFront the numbers of requests and their distribution across different types of requests (like read and write) for your bucket.

Brettski-AWS · Answer

GIven that this is an intermittent issue I recommend that you create a support case and get the support team to dig into why you're seeing failures. Make sure you include request ids as well as times/dates.

Intermittent issues accessing S3 assets through CloudFront

Add your answer

Relevant content