How can I verify an Amazon S3 lifecycle configuration rule for cleaning up incomplete multipart uploads?

Resolution

Note: When you use the AWS Command Line Interface (AWS CLI) to set the rule, the rule is called AbortIncompleteMultipartUpload.

You can verify the lifecycle configuration rule in one of the following ways. 

Query server access logs

To query server access logs, you must have server access logging enabled on your bucket before the lifecycle rule is set to run. After the logs are available, you can review the logs to verify if the rule cleaned up an incomplete multipart upload. For more information on the operations that server access logs report on, see Lifecycle and Logging.

Test the rule by uploading some parts of a multipart upload using the AWS CLI

If you don't have server access logging enabled, then you can test the rule by running an incomplete multipart upload:

Note: The following example uses the AWS CLI to perform a multipart upload. To use an AWS SDK to perform a multipart upload, see Uploading and copying objects using multipart upload.

1. Split the file that you want to upload into multiple parts. For example, if you're using a Linux operating system, you can run the split command, similar to the following:

   split /path/to/filetoupload

2. Run the create-multipart-upload command using the AWS CLI to initiate a multipart upload, similar to the following:

   aws s3api create-multipart-upload --bucket awsexamplebucket --key large_test_file

3. The command returns an output that contains the UploadID. Copy the UploadID value as a reference for later steps. The command output is similar to the following:

   {
       "AbortDate": "Mon, 03 Jun 2019 00:00:00 GMT",
       "AbortRuleId": "multipartcleanup",
       "Bucket": "awsexamplebucket",
       "Key": "objectname",
       "UploadId": "exampleTUVGeKAk3Ob7qMynRKqe3ROcavPRwg92eA6JPD4ybIGRxJx9R0VbgkrnOVphZFK59KCYJAO1PXlrBSW7vcH7ANHZwTTf0ovqe6XPYHwsSp7eTRnXB1qjx40Tk"
   }

4. Run the upload-part command to upload the first part of the file, similar to the following. Replace all values with the values for your bucket, file, and multipart upload:

   aws s3api upload-part --bucket awsexamplebucket --key large_test_file --part-number 1 --body large_test_file.001 --upload-id exampleTUVGeKAk3Ob7qMynRKqe3ROcavPRwg92eA6JPD4ybIGRxJx9R0VbgkrnOVphZFK59KCYJAO1PXlrBSW7vcH7ANHZwTTf0ovqe6XPYHwsSp7eTRnXB1qjx40Tk

5. The command returns an output that contains an ETag value for the part of the file that you uploaded. Copy the ETag value as a reference for later steps. The command output is similar to the following:

6.  Repeat steps 3 and 4 for some parts of the file. For this test, don't upload all parts to complete the file.

   {
       "ETag": "\"example8be9a0268ebfb8b115d4c1fd3\""
   }

   Note: Be sure to increase the part number with each new part that you upload.

7. For this test, don't complete or abort the multipart upload operation. Instead, note the parts that were uploaded by running the list-parts command, similar to the following:

   aws s3api list-parts --bucket awsexamplebucket --key large_test_file --upload-id exampleTUVGeKAk3Ob7qMynRKqe3ROcavPRwg92eA6JPD4ybIGRxJx9R0VbgkrnOVphZFK59KCYJAO1PXlrBSW7vcH7ANHZwTTf0ovqe6XPYHwsSp7eTRnXB1qjx40Tk

8. The command returns a list of the parts, similar to the following:

   {
       "Parts": [
           {
               "PartNumber": 1,
               "LastModified": "2019-06-01T18:17:39.000Z",
               "ETag": "\"example8be9a0268ebfb8b115d4c1fd3\"",
               "Size": 162641
           },
           {
               "PartNumber": 2,
               "LastModified": "2019-06-01T18:18:19.000Z",
               "ETag": "\"example246e31ab807da6f62802c1ae8\"",
               "Size": 3961
           }
       ],
       "Initiator": {
           "ID": "arn:aws:iam::111122223333:user/jane",
           "DisplayName": "jane"
       },
       "Owner": {
           "DisplayName": "bucketowner",
           "ID": "examplea2395fe1985ffabfe0c17d3522e5bc7fa1a2d048f8fc764d7709da80d"
       },
       "StorageClass": "STANDARD"
   }

9. Note the multipart uploads that are in progress for your bucket by running the list-multipart-uploads command, similar to the following:

   aws s3api list-multipart-uploads --bucket awsexamplebucket

   The command returns a list of in-progress multipart uploads, similar to the following:

   {
       "Uploads": [
           {
               "UploadId": "exampleTUVGeKAk3Ob7qMynRKqe3ROcavPRwg92eA6JPD4ybIGRxJx9R0VbgkrnOVphZFK59KCYJAO1PXlrBSW7vcH7ANHZwTTf0ovqe6XPYHwsSp7eTRnXB1qjx40Tk",
               "Key": "large_test_file",
               "Initiated": "2019-06-01T17:08:33.000Z",
               "StorageClass": "STANDARD",
               "Owner": {
                   "DisplayName": "bucketowner",
                   "ID": "examplea2395fe1985ffabfe0c17d3522e5bc7fa1a2d048f8fc764d7709da80d"
               },
               "Initiator": {
                   "ID": "arn:aws:iam::111122223333:user/jane",
                   "DisplayName": "jane"
               }
           }
       ]
   }

10. When you set the lifecycle configuration rule, you specified how many days after the start of a multipart upload the cleanup should occur. Wait the number of days that you set for the rule.

11. Run the list-parts command again to confirm that the parts of the incomplete multipart upload have been deleted. After the parts are deleted by the rule, the command returns this response: "An error occurred (NoSuchUpload) when calling the ListParts operation: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed."

12. Run the list-multipart-uploads command again to confirm that the multipart operation was stopped. After the multipart operation is stopped by the rule, the command returns no output.

Related information

How do I use the AWS CLI to upload a large file in multiple parts to Amazon S3?