Behavior with overlapping regular object and MPU object

Question

I am trying to perform following steps.

Create a bucket
Put an object a_1 (size is 20)

# ./put.py -S amazon -B r1bn-43452 -b 20 a_1
Using 'amazon' bucket 'r1bn-43452' and user '0'

wrote: 20 bytes - 100%   Key a_1 => 20 bytes "89d9f238d9fd1e78dd26ea5427ec212e" None

Initiate mpu a_1 //same name as object uploaded in previous step

# ./mpu.py -S amazon -B r1bn-43452 -i a_1
Using 'amazon' bucket 'r1bn-43452' and user '0'

MPU Initiated a_1 (hURNy_Bv6Bj2T7ro.Ppyt66vhjCZ91_VMnqCJsTswiTUWxSM.sCJlE2lZOFqABYpxMVrJ10fAzlB1kL0uy3NOjmtT9KlKxe2i2IavbuIDBUC1OmR5cAIbykTfUu0goUA) None

Upload 1 part(size is 5000) for mpu a_1

# ./mpu.py -S amazon -B r1bn-43452 -u hURNy_Bv6Bj2T7ro.Ppyt66vhjCZ91_VMnqCJsTswiTUWxSM.sCJlE2lZOFqABYpxMVrJ10fAzlB1kL0uy3NOjmtT9KlKxe2i2IavbuIDBUC1OmR5cAIbykTfUu0goUA -p 1 -k 1 -b 5000 a_1
Using 'amazon' bucket 'r1bn-43452' and user '0'

a_1 hURNy_Bv6Bj2T7ro.Ppyt66vhjCZ91_VMnqCJsTswiTUWxSM.sCJlE2lZOFqABYpxMVrJ10fAzlB1kL0uy3NOjmtT9KlKxe2i2IavbuIDBUC1OmR5cAIbykTfUu0goUA part 1

Delete object a_1

# ./del.py -S amazon -B r1bn-43452 a_1
Using 'amazon' bucket 'r1bn-43452' and user '0'

deleting key a_1:None

List the objects as well as incomplete-mpu //a_1 exist as incomplete-mpu

# ./list.py -S amazon -B r1bn-43452
Using 'amazon' bucket 'r1bn-43452' and user '0'

IsTruncated: False None

# ./mpu.py -S amazon -B r1bn-43452 -l
Using 'amazon' bucket 'r1bn-43452' and user '0'

KY: a_1 hURNy_Bv6Bj2T7ro.Ppyt66vhjCZ91_VMnqCJsTswiTUWxSM.sCJlE2lZOFqABYpxMVrJ10fAzlB1kL0uy3NOjmtT9KlKxe2i2IavbuIDBUC1OmR5cAIbykTfUu0goUA 2024-09-13T19:57:26.000Z arn:aws:iam::521232295589:user/stewari STANDARD None
Truncated: False a_1 hURNy_Bv6Bj2T7ro.Ppyt66vhjCZ91_VMnqCJsTswiTUWxSM.sCJlE2lZOFqABYpxMVrJ10fAzlB1kL0uy3NOjmtT9KlKxe2i2IavbuIDBUC1OmR5cAIbykTfUu0goUA

Complete mpu a_1 //complete-mpu is successful

# ./mpu.py -S amazon -B r1bn-43452 -u hURNy_Bv6Bj2T7ro.Ppyt66vhjCZ91_VMnqCJsTswiTUWxSM.sCJlE2lZOFqABYpxMVrJ10fAzlB1kL0uy3NOjmtT9KlKxe2i2IavbuIDBUC1OmR5cAIbykTfUu0goUA -c a_1
Using 'amazon' bucket 'r1bn-43452' and user '0'

MPU Completed a_1 "c472042aba996f8130919de65e9d88bb-1" None

List the objects as well as incomplete-mpu //no results

# ./list.py -S amazon -B r1bn-43452
Using 'amazon' bucket 'r1bn-43452' and user '0'

IsTruncated: False None

# ./mpu.py -S amazon -B r1bn-43452 -l
Using 'amazon' bucket 'r1bn-43452' and user '0'

Truncated: False

Head-object/ Get-object fails with 404 error //specified key a_1 does not exist

# ./get.py -S amazon -B r1bn-43452 a_1
Using 'amazon' bucket 'r1bn-43452' and user '0'


=====> HEAD 404 and GET 404 both failed <======

S3ResponseError: 404 Not Found
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>a_1</Key><RequestId>R9Q1NJTBAWP9R3Z7</RequestId><HostId>QUXzkJ/ByWd9+Z+Cio+74IlKnTVCtPAvGxjT1m/enOhKvehvdBhwL2Oty+/ixJTXy8/Vvxp7QBI=</HostId></Error>

After deleting the object in step 5 above, S3 lists incomplete-mpu (step 6) and also performs complete-mpu successfully in step 7. However it does not list any object in step 8. Since S3 does not list anything is step 8, head-object/ get-object fails in step 9. This behavior is conflicting since it performs complete-mpu successfully but deletes it later. Either step 8 should list object (mpu) or step 6/ 7 should fails.

I expected S3 to return MPU object in step 8. Is this an issue or is this behavior expected ? If the behavior is correct, can anyone help in understanding this behavior ?

Hari_Priya_G · Answer

Hello,

I've followed the steps mentioned by you at my end and noticed the following behaviour:

1. When versioning is in disabled state for S3 bucket, then there's no object in the bucket after the completion of multipart upload. This is similar to the behaviour observed at your end.

2. When versioning is in enabled state for S3 bucket and

a) when the regular object is temporarily deleted (deletion without including version ID) in step 5, then, after the completion of multipart upload, the MPU object is stored as a noncurrent version in the bucket alongside the regular object while the delete marker for the object is stored as the current version.

b) when the regular object is permanently deleted (deletion by including version ID) in step 5, then, after the completion of multipart upload, the MPU object is stored as a current version in the bucket with no noncurrent versions.

The above behaviours are expected as mentioned in AWS public documentation https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html#distributedmpupload

From the documentation:

"For buckets that don't have versioning enabled, it is possible that some other request received between the time when a multipart upload is initiated and when it is completed might take precedence."

"For example, if another operation deletes a key after you initiate a multipart upload with that key, but before you complete it, the complete multipart upload response might indicate a successful object creation without you ever seeing the object."

"When the buckets have S3 Versioning enabled, completing a multipart upload always creates a new version."

Behavior with overlapping regular object and MPU object

Add your answer

Relevant content