- Newest
- Most votes
- Most comments
You can implement S3 Lifecycle rule, either delete or move to a different tier S3 storage.
See Optimize storage costs with new Amazon S3 Lifecycle filters and actions
To identify and delete files in your S3 bucket that haven't been accessed or retrieved in a specified period, you can use a combination of S3 features and best practices:
-
Enable S3 Server Access Logging: This will help you track object access patterns. You can configure your S3 bucket to log all access requests, including GET requests that indicate when an object was last retrieved.
-
Analyze Access Logs: Once you have access logs, you can analyze them to identify objects that haven't been accessed within your specified timeframe. You might need to use additional tools or scripts to process these logs effectively.
-
Use S3 Lifecycle Rules: After identifying unused objects, you can set up S3 Lifecycle configuration rules to automatically delete these objects. While Lifecycle rules don't directly use access patterns, you can use them in conjunction with your analysis to manage object deletion.
-
Consider using S3 Analytics: This feature can help you analyze storage access patterns and transitions, which might assist in identifying less frequently accessed objects.
-
Implement Custom Solution: For more precise control, you might need to develop a custom solution that combines log analysis with programmatic deletion using the S3 API.
Remember to be cautious when deleting objects, especially in a production environment. It's recommended to first move potentially unused objects to a different storage class (like S3 Glacier) before permanent deletion.
Also, keep in mind that while this approach can help manage storage and reduce costs, it requires careful implementation to avoid unintended deletions. Always ensure you have proper backups and thoroughly test your deletion process before applying it to your main website's picture database.
Sources
Deleting Amazon S3 log files - Amazon Simple Storage Service
Deleting Amazon S3 objects - Amazon Simple Storage Service
Deleting a single object - Amazon Simple Storage Service
answered 2 years ago
I order to delete S3 objects based on access pattern you need to log object level API calls (by default AWS do not log S3 object level access details). There are two possible ways to do log that -
- Server Access logs
- Cloud trail data events logs
-- Server access logs has no additional cost of ingestion, only the storage cost of the access logs that goes to user/customer's bucket.
-- Cloud trail data events has a cost of ingestion & storage.
In addition to S3 access API calls you will need to configure S3 inventory and ingest the report into an S3 bucket. Now when you have both of these data, you have to create Amazon Athena tables.
To query this data from Athena you will ultimately need a Lambda function to query last access date of all the objects and delete if it is find older than your specified date. You can add a tag also to identified objects and further delete them via S3 LC policy. You can schedule this lambda function daily or as desired.
This blog shows high-level architecture to identify objects, tag them and delete via LC policy, though I am surprise it has no detail on lambda code and Athena query.
Pls note that if your customer has no access logs or cloud trail data events for past, they have to be very careful in deletion because query may show an object is not accessed in past month/year if it has no last access time stamp for past period.
Relevant content
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 9 months ago
S3 lifecycle can only delete objects based on age, it cannot expire or transition objects based on access pattern.