By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

how to empty version enabled bucket faster?

0

While doing the bucket empty using below aws s3 cli, it removed the objects and now for current version is it not showing any object but when I see using version checkbox enabled it shows more object, so I need to empty other version's object as well other wise bucket deleton is not possible, please let me know hot to do it in faster way.

Cli used for emptying the bucket.

aws s3 rm s3://<bucketName>--recursive

I have also enabled the lifecycle on the bucket but does any command also needed to remove the object

Topics
StorageManagement & Governance
Tags
Amazon Simple Storage ServiceAmazon S3 GlacierAWS Command Line InterfaceStorageS3 Object Lock
Language
English
rePost-User-9210655
asked a year ago280 views
2 Answers
1

Hi,

You can try to use a combination of delete-objects CLI command with a list-object-versions

aws s3api delete-objects --bucket <bucketName> \ 
  --delete "$(aws s3api list-object-versions \
  --bucket "<bucketName>" \
  --no-paginate  \
  --query='{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

Hope it helps ;)

profile picture
EXPERT
Antonio_Lagrotteria
answered a year ago
  • rePost-User-9210655
    a year ago

    Thanks @alatech

    Seems it is working, but it is listing array of 10 objects only, saying those 10 objects are deleted, so does it need to call in loop untill all objects get deleted?

    Let me try.

  • Antonio_Lagrotteria EXPERT
    a year ago

    Try tweaking the parameters related to pagination: https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html

  • autarch
    11 days ago

    This is extremely slow though, and sadly not parallelisable, due to the terrible way AWS SDK/CLI handles pagination. Lifecycle rules are so far the best method I have found, but even that takes days just do to the delay with a minimum of 1 day of age for the two steps each, and more if there are a large number of objects. I've tried S3 Batch Operations, but not only is it very hard to operate, but it doesn't seem to support deleting actual objects or worse object versions, just tags of objects or copying objects.

0

While running suggested "aws s3api delete-objects ..." command.

I observed the matrics of the bucket, it shows the size increased by few more GB, and can see every (almost all) objects in the bucket are marked as delete marker.

Wants to understand how delete is working, is it like first it will marke object to delete marker and then in coming future those objects will get deleted?

Will it get deleted by lifecycle set on the bucket?

rePost-User-9210655
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content