After performing a large deletion operation in Amazon S3, the list command is unresponsive. How do I troubleshoot this?

3분 분량

I've performed a large number of file deletions and I'm trying to run the list command on my Amazon Simple Storage Service (Amazon S3) bucket. However, the list command hangs and doesn't return any output. How do I resolve this?

Short description

If you attempted to perform a large number of file deletions and the ls or list-objects command hangs, try the following troubleshooting tips:

Remove expired object delete markers.
Try the list-objects-v2 command or the latest version of the AWS Command Line Interface (AWS CLI).
Use an Amazon S3 inventory report.

Resolution

Remove expired object delete markers

In a versioning-enabled bucket, when a delete operation is made for a versioned object (without specifying a version ID), then a delete marker is created. When previous object versions are set to expire using lifecycle configuration rules, delete markers that are associated with the expired objects are in the bucket.

If there are thousands of delete markers, the list operation might timeout. Therefore, it's a best practice to remove the expired object delete markers from the bucket to improve the performance of a list operation.

You can use a lifecycle rule to clean up expired object delete markers:

1. Open the Amazon S3 console.

2. From the list of buckets, choose the bucket that has the expired object delete markers.

3. Choose the Management tab.

4. Choose Add lifecycle rule and enter a rule name. Then, choose Next.

5. For Storage class transition, you can skip this section by choosing Next.

6. For Configure expiration, select Clean up expired object delete markers. Then, choose Next.

7. Review the lifecycle rule settings, and then choose Save.

Try the list-objects-v2 command or the latest version of the AWS CLI

The list-objects-v2 command (or ListObjectsV2 API call) is an updated version of the list operation that includes some performance improvements. The following example command requests a list of objects in awsexamplebucket:

aws s3api list-objects-v2 --bucket awsexamplebucket

Additionally, confirm that you're using the latest version of the AWS CLI. Earlier AWS CLI versions might still be using previous ListObjects API call.

Note: For backward compatibility, Amazon S3 continues to support the ListObjects API call.

Use an Amazon S3 inventory report

Instead of running a list operation, you can use an Amazon S3 inventory report to get a list of your objects. When a bucket has hundreds of millions of objects, running multiple list operations (listing 1,000 objects at a time) can be challenging.

You can get daily or weekly Amazon S3 inventory reports for a bucket or prefix, and you can include additional object metadata in your reports. Additionally, you can pass an inventory report as an input for a batch operation, and you can use Amazon Athena to query a report.