looking for automation to Delete S3 files with specific Extension and date
I have Automation to delete files with specific range of dated files, but looking for Automation if we have any on Deleting s3 files with specific Extensions and data together which are 60 days old.
Lifecycle is a great mechanism within S3 to automatically delete files based on certain criteria. You can specify a filter by using object size, object key prefix, one or more object tags, or a combination of filters. So one way would be to tag your objects when they are created/uploaded and apply the lifecycle policy. https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html
Another way would be utilize S3 Inventory report. Identify objects that you want to delete and leverage SDKs/API's as show below to submit request for multiple deletes. https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-multiple-objects.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
You can also use Athena queries to identify the objects from s3 inventory. https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-athena-query.html
I built this and working for my requeirment #!/bin/bash
SINCE_today=date --date '-13 weeks -1 days' +%F 2>/dev/null || date -v '-2w' -v '+2d' +%F
bucket=POC_DEV_Bucket
aws s3api list-objects --bucket "$bucket" --query 'Contents[?LastModified < '"$SINCE_today"']' --output text > 90days_old.txt && grep -i ".json" 90days_old.txt > s3file_with_extension_with_daysold.txt && cat s3file_with_extension_with_ZERO_daysold.txt && awk '{$1= ""; print $2}' s3file_with_extension_with_daysold.txt | xargs -I {} aws s3 rm s3://"$bucket"/{}
I have Lifecycles enabled, but some bucket which has huge data in terabytes need to be deleted by using particular extension like .json,.csv,.html with 90 days old.
so wondering is there a lambda or some automation to do that before I needed to work on new code
Relevant questions
S3 Intelligent - Tiering - 'Monitoring and Automation' costs
asked 2 years agoMove files from S3 to FSx for Windows
Accepted Answerasked 2 years agoHow to delete old files deployed in S3 with CodePipeline
asked 2 months agoIs there an option in 'My Files' for a user to delete files?
asked 3 years agoHow can we set s3 lifecycle rule to delete the files form S3 bucket!
asked 2 months agoCan I delete AWS Systems Manager Automation Execution History entries?
asked 9 days agoLambda Automation
asked 6 months agolooking for automation to Delete S3 files with specific Extension and date
Accepted Answerasked 23 days agoS3 for backup of large small files on-premise
Accepted Answerasked 2 years agoHow do I restore deleted files in S3?
asked 4 months ago