By using AWS re:Post, you agree to the Terms of Use
/looking for automation to Delete S3 files with specific Extension and date/

looking for automation to Delete S3 files with specific Extension and date

0

I have Automation to delete files with specific range of dated files, but looking for Automation if we have any on Deleting s3 files with specific Extensions and data together which are 60 days old.

3 Answers
3

Lifecycle is a great mechanism within S3 to automatically delete files based on certain criteria. You can specify a filter by using object size, object key prefix, one or more object tags, or a combination of filters. So one way would be to tag your objects when they are created/uploaded and apply the lifecycle policy. https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html

Another way would be utilize S3 Inventory report. Identify objects that you want to delete and leverage SDKs/API's as show below to submit request for multiple deletes. https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-multiple-objects.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html

You can also use Athena queries to identify the objects from s3 inventory. https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-athena-query.html

answered 23 days ago
0
Accepted Answer

I built this and working for my requeirment #!/bin/bash

SINCE_today=date --date '-13 weeks -1 days' +%F 2>/dev/null || date -v '-2w' -v '+2d' +%F

bucket=POC_DEV_Bucket

aws s3api list-objects --bucket "$bucket" --query 'Contents[?LastModified < '"$SINCE_today"']' --output text > 90days_old.txt && grep -i ".json" 90days_old.txt > s3file_with_extension_with_daysold.txt && cat s3file_with_extension_with_ZERO_daysold.txt && awk '{$1= ""; print $2}' s3file_with_extension_with_daysold.txt | xargs -I {} aws s3 rm s3://"$bucket"/{}

answered 16 days ago
0

I have Lifecycles enabled, but some bucket which has huge data in terabytes need to be deleted by using particular extension like .json,.csv,.html with 90 days old.

so wondering is there a lambda or some automation to do that before I needed to work on new code

answered 19 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions