- Newest
- Most votes
- Most comments
Hello.
Since you cannot delete objects including versions in bulk using the AWS CLI, you must delete them by combining commands.
I tried it with my AWS account, and was able to delete objects under a specific folder, including their versions, using the command below.
In the example command below, all objects under "file-path/" will be deleted.
The commands below can be executed using CloudShell.
https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html
aws s3api delete-objects --bucket example-bucket --delete "$(aws s3api list-object-versions --bucket example-bucket --prefix file-path/ | jq '.Versions + .DeleteMarkers | {Objects: map({Key, VersionId})}')"
If you don't absolutely need the objects to be deleted instantly, I'd generally advise using S3 lifecycle rules to delete them. It scales to any number of objects and takes care of all the versioning and delete marker complexities without having to think about expressing the intent on the command line.
First create a lifecycle rule and set it to apply to objects with the folder name as the prefix filter. Configure these two actions for the rule:
- "Expire current versions of objects": 1 day after object creation
- "Permanently delete noncurrent versions of objects": 1 day after objects become noncurrent (leave the "number of newer versions to retain - optional" field empty)
Create a second lifecycle rule that applies to all objects in the bucket. Enable these two actions:
- "Delete expired object delete markers": enabled
- "Delete incomplete multipart uploads": 3 days (appropriate for typical workloads, but consider for your own use case)
Lifecycle rules are executed starting at midnight UTC every day. On the first day after you've created the rule, the current versions will become noncurrent, with only new delete markers getting created. A day later, versions prior to the delete markers will get deleted permanently. Finally, the delete markers that no longer cover any prior object versions will get deleted.
For a moderately sized bucket, all those actions may complete in the first three days, but if the number of objects is large, a small percentage of objects may be skipped, waiting until the next day to be retried.
Relevant content
- asked 7 months ago
- AWS OFFICIALUpdated a month ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 3 months ago
- AWS OFFICIALUpdated 2 days ago
Hi,
Thank you for your answer. Does this also work without CloudShell, from my own work environment? Because I need to connect to an internal endpoint-url.
Kind regards,
Laurens
Yes, it can be used if it can be executed from a bash shell etc. where the "jq" command can be used. Also, as @Leo K explains, if there is no urgent need to delete it, I think you can delete it by setting a life cycle rule.