Automatically tag S3 objects by prefix or help with lifecycle rules

Question

We have large amounts of data written to S3 via an S3 file gateway SMB share.   Each day's data lands in a new folder/prefix with date dynamically added to the folder name.  Some folder data we want to keep for 100 days and purge.  Other folder data we want to keep for a year and purge.  While we are at it, we should probably purge empty prefixes/folder names.   The application writing the data to the S3 gateway can add text to the start or end of the folder name automatically but apparently lifecycle rules can't use wildcards in prefix names.

In the scenario below we want to keep data written in  Weekly* prefixes for 100 days and Keep* prefixes for 365 days.  Research on the subject mostly focused on tagging which would work great but I don't know how to tag dynamically by text in a prefix name.  Lifecycle rules would be easy with these tag examples but don't want to make a manual tagging process:

**Weekly*** were tagged with key=retention, value 100

and

**Keep*** key=retention, value 365.

\SomeRoot\
                                 Weekly-2023-09-07\
                                                                      FileA.txt
                                                                      FileB.txt 
                                                                      etcFile
                                 Weekly-2023-09-14\
                                 Weekly-2023-09-21\
                                 Keep-2023-09-28\
                                 Weekly-2023-10-05\
                                 Weekly-2023-10-12\
                                 Weekly-2023-10-19\
                                 Keep-2023-10-26\
                                 Weekly-2023-11-02\
                                 EtcPrefix…

Objects written in \SomeRoot and \SomeRoot\* will be moved to Glacier Immediate Access after 7 days. This part seems easy enough but wanted to mention it

Answer

Hi,

You'll see examples of lifecycle rules on this page: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html like this one

```

Transition and Expiration Rule
    
    Enabled
    
      365
      S3 Glacier Flexible Retrieval

3650

```

The rules are heavily based on prefix. So, I would suggest a slight change in your directory structure:
```
/keep100/[data to keep for 100 days structure]
/keep365/[data to keep for 100 days structure]
```
Then you can easily apply the  xml node that you need in a lifecycle applied to the distinct prefixes.

Best

Didier

Automatically tag S3 objects by prefix or help with lifecycle rules

相關內容