Optimizing Storage Costs by Transitioning Millions of S3 Objects from Standard to Glacier Tier
For many companies, controlling storage costs associated with large object stores in AWS S3 Standard tier can be challenging. This article discusses how have helped one customer transitioned over 250 million objects from S3 Standard to Glacier tier more cost effectively using lambda functions and file concatenation techniques.
The Challenge: The customer had accumulated over 250 million individual objects stored in S3 Standard tier over time. While migrating to Glacier tier would reduce ongoing storage costs, directly transitioning such a large number of objects would result in prohibitively high lifecycle configuration charges.
The Solution: Rather than attempting a direct migration, the customer implemented a two-step approach using AWS Lambda. First, Lambda functions were used to compress and concatenate multiple source objects into single "bundle" files. This reduced the total number of objects that would require lifecycle actions. To improve performance and reduce data transfer costs, the Lambda functions accessed the S3 buckets directly using S3 PrivateLink. Once file bundles had been created, lifecycle rules were configured to transition the consolidated objects to Glacier tier. By bundling 20 files at a time into single objects, estimates showed the total number of objects requiring transition could be reduced by two orders of magnitude. This led to predicted monthly savings of over $2,700 compared to remaining in Standard tier, even after accounting for Lambda processing costs.
Key Takeaways: For companies managing very large S3 object stores, file compression and concatenation techniques using Lambda can enable more cost-effective archiving to Glacier tier. Leveraging services like PrivateLink/S3 vpc endpoints can further optimize performance and costs. With proper planning and testing, these approaches allow achieving substantial long-term storage savings that offset initial migration costs.
- AWS Documentation on Amazon S3 Batch Operations: https://aws.amazon.com/blogs/storage/data-management-at-scale-using-amazon-s3-batch-operations/
- S3 Utilities open source project for multipart concatenation: https://github.com/whitfin/s3-utils
- AWS Calculator for estimating Lambda function costs: https://calculator.aws/#/createCalculator/Lambda
Relevant content
- Accepted Answerasked a month ago
AWS OFFICIALUpdated 6 months ago
