CloudFront resources fail to load from South Africa ONLY due to CORS headers missing

Question

I've received complaints from users of my site hosted using AWS resources that it is not loading from South Africa. Upon checking their HAR files, and installing South Africa VPNs, I have found out that there is a **single javascript file** that does not have the CORS headers served when the client is from South Africa!

Here is the cURL request/response when my laptop is connected to South Africa VPN:

```
❯ curl 'https://files.developerhub.io/main-E2LOABSB.js' -I --compressed
HTTP/2 200
content-type: application/javascript
content-length: 3706182
date: Thu, 18 Jul 2024 15:34:23 GMT
last-modified: Thu, 18 Jul 2024 15:16:58 GMT
etag: "65b9e90c9cbbfb3dcc8cbeb2ad126589"
x-amz-server-side-encryption: AES256
cache-control: max-age=604800,public
accept-ranges: bytes
server: AmazonS3
vary: Accept-Encoding
x-cache: Hit from cloudfront
via: 1.1 cc087bb8398c8b9dbcab903c425edc8c.cloudfront.net (CloudFront)
x-amz-cf-pop: CPT52-C1
alt-svc: h3=":443"; ma=86400
x-amz-cf-id: JEqxPpRcTGpg2QCsSlhi5g-JAi_2kvuK8OCbCxIf7sObn50M9Sv69g==
age: 355387
```

And when the VPN is disconnected or connected to UK, US, Eur, Australia, or any other country I've tested:

```
❯ curl 'https://files.developerhub.io/main-E2LOABSB.js' -I --compressed
HTTP/2 200
content-type: application/javascript
date: Thu, 18 Jul 2024 15:22:16 GMT
access-control-allow-origin: *
access-control-allow-methods: GET, HEAD
access-control-max-age: 3000
last-modified: Thu, 18 Jul 2024 15:16:58 GMT
etag: W/"65b9e90c9cbbfb3dcc8cbeb2ad126589"
x-amz-server-side-encryption: AES256
cache-control: max-age=604800,public
server: AmazonS3
content-encoding: gzip
vary: Accept-Encoding
x-cache: Hit from cloudfront
via: 1.1 1976c726f5a49e79daf18d11f7fa62da.cloudfront.net (CloudFront)
x-amz-cf-pop: MRS52-C2
alt-svc: h3=":443"; ma=86400
x-amz-cf-id: hDzv1mEGVVzHAmKMiCe3E2RMEaIbr2_0ceZWi9OCEuFPZ0tJ--ykCw==
age: 361581
```

All other javascript/CSS files are getting served fine with CORS, and they exist in the same S3 bucket, and the same CloudFront distribution. The CloudFront distribution has no geo restrictions. I've checked the headers set from the S3 origin on the file, and they're no different from any other file "Cache-Control: max-age=604800,public" and "Content-Type: application/javascript".

To debug further, I enabled standard CloudFront logging, and unusually I found out that when those files are requested by client's browser from South Africa (and fail due to CORS error, Allow-Control-Allow-Origin not present), the `sc-content-len` is set. None of the other files that get served have `sc-content-len` set.

I have resolved the issue of the site not loading by deploying my application again, so it's not currently using this file anymore, but this to me seems like a one-off error happening on AWS side.

Any ideas what could've gone wrong? Is it really a one-off exception?

zed · Accepted Answer

Thanks for pointing me in the right direction that every CloudFront edge location has its own cache and could cache the response differently.

Upon invalidation, and attempting to get the file from UK client without adding `-H "Origin: https://origin.com` to the cURL request, I realized that the file got cached in the edge closest to me **without** the CORS headers. There was no way to fix this issue until I re-invalidate. Thus, I realised that the original file `main-E2LOABSB.js` could've been requested by a non-browser in South Africa before it was requested by a browser, which caused the CORS headers to be missing for that file.

Upon this, I checked the Behaviour settings for my CloudFront distribution, and I realized that I am still using legacy settings since I first set up the distribution when the current new settings did not exist.

I modified my legacy settings to a custom cache policy, and to use Managed-CORS-S3Origin with Managed-CORS-with-preflight-and-SecurityHeadersPolicy.

Now after invalidating the file, when Origin header is present, the CORS headers show. If the Origin header is not present, the CORS headers do not show. All disregarding to order of which request was first. This resolves my problem.

Yaniv Rozenboim · Answer

One possible reason is that the object was cached in South Africa's CloudFront PoP before you added the CORS related headers in your origin (S3).

If you added the CORS header after the object was already cached by CloudFront it will not be automatically propagated and you need to manually invalidate the cache entry (or wait for the object to expire).

e.g.,
```aws cloudfront create-invalidation --distribution-id YOUR_DISTRIBUTION_ID --paths /main-E2LOABSB.js```

CloudFront resources fail to load from South Africa ONLY due to CORS headers missing

Relevant content