Check if files exist in S3 without read permissions

Question

Hello,

We're building a service that uploads data that it collects from various devices in the wild.
The service is running on the device, and we do NOT want it to be able to read data - just upload. 
However, we wouldn't want it to re-upload data that is already in the S3.
We tried to use HeadObject, and put the file's md5 in the header, however it does not seem possible to give HeadObject permission without GetObject.

What other possible solutions are there?

Thank you!

Answer

I'm not aware of any way to do this directly with the S3 API. I think you'll have to create your own lightweight API on top, like with an API Gateway and a Lambda function that essentially proxies the S3 API using a role with GetObject permissions, and that only supports the HeadObject action.

Depending on the specifics of your setup, it may be possible to do without the Lambda, using S3 at the integration endpoint directly for the API Gateway method.

Answer

Hi,

Good question.  If you are able to check the objects and their metadata without reading content, would that be enough?

A couple different solutions:
* Tagging the object with metadata (device, other information you have) and ensuring it's not re-writing.
* Only using ListObjects (ListObjectsV2) and non-object read permissions (GetObject).  Keep in mind that ListObjectsV2 requires READ access to the Bucket.

You're right about HeadObject!  The HeadObject action retrieves metadata from the object without returning the object itself.  However, this requires READ Access to the object.    (https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html).

Thus, if you use a combination of the IAM Actions: ListObjects and ListBucket, GetObjectTagging, GetObjectACCL (if needed) and Deny GetObject - that should take care of your use case!

AWS S3 Actions: https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html

Answer

There is a ListObjects and ListObjectsV2 actions against S3. These actions may need read access, but you might be able to determine file existence with 1 of these calls and maybe a combination of resource-based (in s3) and IAM based policies that allow read, but not GetObject.

Check out this to test - https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html - ListObjectsV2 mentions that "Read access is needed", am not sure if read access implies "GetObject" action, or some other way of configuring it.

Alternatively, you could try using GetObjectAcl - that might allow you to determine if a key exists without getting access to the object itself.

Check if files exist in S3 without read permissions

관련 콘텐츠