how to set the s3 lifecycle policy to auto-delete files and preserve folder in a bucket

S3 lifecycle policies don't allow you to expire using a suffix (.wav), that means that if you want to expire all objects that have .wav extension you can't use S3 lifecyle as that it's not possible.

When you setup an S3 lifecycle policy you need to use the prefix or Tag, means you can create a rule for the prefix:

sample/ 

This rule will expire all objects under that "folder", in other words, expire all S3 objects that start with "sample/" (sample/ included). If there are no other objects under that prefix, S3 will no longer display the "folder", keep in mind, a folder on S3 it's just a zero byte object with the delimiter "/" on its name.

Example:

$ aws s3api put-object --bucket test --key folder/

The above AWS Cli command will create a zero byte object named "folder/", which S3 will display as "folder" due to the delimiter "/".

For you to expire all objects under that prefix and still keep the "folder", first you will to create a "folder" with that name (the empty object with the delimiter), so that an object with that name (folder/) exists on S3, and second, a lifecycle rule for each possible option, this is, one rule for each possible start of filename (you can have to 1000 lifecycle rules):

sample/a
sample/b
...
sample/z

The above rules, will expire all objects that start with "sample/a" ... to "sample/z" but not expire the prefix "sample/" (your empty object), and so preserving the "folder" even so there are no further objects under that prefix.

Object deletion based on suffix (*.wav) is not possible through S3 lifecycle. Only option is AWS CLI/SDK script / Lambda function. It will constitute a LIST API first before deletion, if number of objects are less, then this additional LIST cost won't be an issue.

CLI example - need to check if it will preserve the folder/prefix if it becomes empty. I will check in a while.

aws s3 rm s3://Path/To/Dir/ --recursive --exclude "*" --include "*.wav"

Another possible way is to first to add a tag to all these in-scope objects (*.wav) refer doc and then create a LC rule that's only applicable to objects associated with that tag. Again it will cost LIST API at first place + object tagging cost. If numbers of objects are several hundred millions then cost optimized way is to create an S3 inventory, filter all *.wav object's name, ingest them into a .csv and use this as a manifest to apply tag through S3 batch operation and then LC rule to delete by tag filtering.

For a continuous, automated solution a lambda function to apply tag on *.wav objects based on S3 PUT event would be much appropriate.

Quick test on versioning disabled bucket

[ec2-user@ip-172-16-1-92 ~]$ aws s3 ls  s3://cur-bucket-razguru/sample/
2024-03-07 18:12:06          0 
2024-03-07 18:13:44          460 part.wav
2024-03-07 18:13:22          12850 vix.wav

[ec2-user@ip-172-16-1-92 ~]$ aws s3 ls  s3://cur-bucket-razguru/sample
                           PRE sample/

[ec2-user@ip-172-16-1-92 ~]$ aws s3 rm s3://cur-bucket-razguru/sample --recursive --exclude "*" --include "*.wav"
delete: s3://cur-bucket-razguru/sample/part.wav
delete: s3://cur-bucket-razguru/sample/vix.wav

[ec2-user@ip-172-16-1-92 ~]$ aws s3 ls  s3://cur-bucket-razguru/sample
                           PRE sample/
[ec2-user@ip-172-16-1-92 ~]$ aws s3 ls  s3://cur-bucket-razguru/sample/
2024-03-07 18:12:06          0