CloudFront doesn't honor S3 REST API ListObjectV2 request parameters

Question

While programmatically listing objects of a bucket I noticed that the behavior changes when using CloudFront. When running the ListObjectV2 [1] request against the origin bucket endpoint and using a delimiter parameter, S3 returns a limited set of the objects. When running the same request against the CloudFront distribution endpoint, S3 returns all objects.

Request against bucket endpoint:

```
16:19:09	Command:	GET /?delimiter=/&list-type=2 HTTP/1.1
16:19:09	Command:	Authorization: *****************************************************************************************************************************************************************************************************************
16:19:09	Command:	Connection: keep-alive
16:19:09	Command:	Host: .s3.dualstack.us-east-1.amazonaws.com
16:19:09	Command:	Keep-Alive: 300
16:19:09	Command:	x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
16:19:09	Command:	x-amz-date: 20241120T191909Z 
16:19:09	Response:	HTTP/1.1 200 OK
16:19:09	Response:	x-amz-id-2: HBZPmBRMbKSS4P1MsEVLQuubB5ep5Cr6lZIfoSDj7Ao7B/290uHhbGRYK3ZEWAdynL9FcVR3B8A=
16:19:09	Response:	x-amz-request-id: FMY82W1DFPDWJMYG
16:19:09	Response:	Date: Wed, 20 Nov 2024 19:19:10 GMT
16:19:09	Response:	x-amz-bucket-region: us-east-1
16:19:09	Response:	Content-Type: application/xml
16:19:09	Response:	Transfer-Encoding: chunked
16:19:09	Response:	Server: AmazonS3
16:19:09	Listing:	
16:19:09	Listing:	
16:19:09	Listing:	  EDITED
16:19:09	Listing:	  
16:19:09	Listing:	  1
16:19:09	Listing:	  1000
16:19:09	Listing:	  /
16:19:09	Listing:	  false
16:19:09	Listing:	  
16:19:09	Listing:	    My Directory/
16:19:09	Listing:	  
16:19:09	Listing:	
```

Request against CloudFront endpoint:

```
16:20:13	Command:	GET /?delimiter=/&list-type=2 HTTP/1.1
16:20:13	Command:	Authorization: *****************************************************************************************************************************************************************************************************************
16:20:13	Command:	Connection: keep-alive
16:20:13	Command:	Host: EDITED.cloudfront.net
16:20:13	Command:	Keep-Alive: 300
16:20:13	Command:	x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
16:20:13	Command:	x-amz-date: 20241120T192013Z
16:20:13	Response:	HTTP/1.1 200 OK
16:20:13	Response:	Content-Type: application/xml
16:20:13	Response:	Transfer-Encoding: chunked
16:20:13	Response:	Connection: keep-alive
16:20:13	Response:	Date: Wed, 20 Nov 2024 19:14:21 GMT
16:20:13	Response:	x-amz-bucket-region: us-east-1
16:20:13	Response:	Server: AmazonS3
16:20:13	Response:	X-Cache: Hit from cloudfront
16:20:13	Response:	Via: 1.1 cc532083828b8e3e611a2f0d99536f1c.cloudfront.net (CloudFront)
16:20:13	Response:	X-Amz-Cf-Pop: GRU1-P3
16:20:13	Response:	X-Amz-Cf-Id: Pt_cTRfb7LYRYvmgms9lGgZFi4KZL7qaAetXgj5F0ItNB1cCZjeBWg==
16:20:13	Response:	Age: 353
16:20:13	Listing:	
16:20:13	Listing:	
16:20:13	Listing:	  558007041164
16:20:13	Listing:	  
16:20:13	Listing:

```

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

Leo K · Accepted Answer

Did you set the CloudFront distribution to forward the relevant query string parameters to the origin? The available options at the distribution level are explained in this documentation section: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesQueryString

You can also add query string parameters to origin requests without including them in the cache key with origin request policies: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/origin-request-understand-origin-request-policy.html#origin-request-understand-origin-request-policy-settings

re:Post Agent · Answer

The behavior you're observing is consistent with how CloudFront handles requests to Amazon S3 origins. When you make a ListObjectsV2 request directly to the S3 bucket, S3 processes the request parameters, including the delimiter, as expected. However, when the same request goes through CloudFront, it appears that CloudFront is not passing along or honoring the delimiter parameter in the same way.

This difference in behavior occurs because CloudFront is designed to forward certain types of requests to S3, but it doesn't necessarily process or interpret all S3-specific query parameters in the same way that S3 does directly. CloudFront is primarily designed for content delivery and caching, not for replicating all S3 API functionalities.

When CloudFront receives a GET request with query parameters, it typically forwards the entire request to the S3 origin. However, CloudFront's behavior regarding S3-specific operations like ListObjectsV2 may not align perfectly with direct S3 API calls, especially when it comes to parameters like delimiters that affect how S3 processes and returns results.

It's important to note that CloudFront is optimized for serving and caching individual objects, rather than for listing operations. If you need to perform S3 API operations like ListObjectsV2 with specific parameters, it's generally recommended to do so directly against the S3 endpoint rather than through CloudFront.

If you require both the content delivery capabilities of CloudFront and the full functionality of S3 API operations, you might consider using separate endpoints: CloudFront for content delivery and direct S3 API calls for operations like listing objects with specific parameters.  
**Sources**  
[Request and response behavior for Amazon S3 origins - Amazon CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorS3Origin.html)  
[HTTP 403 status code (Permission Denied) - Amazon CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-403-permission-denied.html)

CloudFront doesn't honor S3 REST API ListObjectV2 request parameters

Add your answer

Relevant content