The Amazon S3 charges on my AWS bill are unexpectedly high. What can I do to reduce my Amazon S3 costs?

Short description

Amazon S3 charges are based on storage, requests and data retrievals, data transfer (including S3 Transfer Acceleration usage), and data management.

To identify where you can cut down on your Amazon S3 charges, you must first understand your current billing and usage for Amazon S3. Then, you must understand the ways that you can incur cost.

After you understand how you're charged on Amazon S3, consider the following tips for reducing your billing costs:

• Clean up incomplete multipart uploads.
• Delete previous versions of objects that you don't need.
• Review your storage-class transition costs.
• Review your data retrieval costs.
• Track the requests made to your bucket.
• Check for changes in the size of your bucket.
• Review the cost of individual buckets.
• Understand how your usage relates to your charges.

You can also use the following Amazon S3 features to help you investigate billing concerns:

• Use Amazon S3 Storage Lens to obtain a single view of usage and activity across your Amazon S3 storage. S3 Storage Lens generates insights at the organization, account, bucket, object, or even prefix level. S3 Storage Lens analyzes these storage metrics to deliver recommendations that can help you optimize storage costs and apply best practices for protecting your data.
• Use Amazon S3 Storage Class Analysis to observe data access patterns. With S3 Storage Class Analytics, you can determine when to transition less frequently accessed STANDARD storage to the STANDARD_IA (IA, for infrequent access) storage class.
  Note: Amazon S3 Storage Class Analysis must be configured on a per bucket basis.

Resolution

Clean up incomplete multipart uploads

After you initiate a multipart upload, Amazon S3 retains all the uploaded parts of the file until you either complete or abort the multipart upload. If you don't complete the multipart upload successfully, then Amazon S3 will continue to store the uploaded parts. As a result, you'll be charged for the storage of uploaded parts.

If you use the AWS Command Line Interface (AWS CLI) to abort the multipart upload (abort-multipart-upload), then the operation deletes the (incomplete) uploaded parts. However, if you use other tools that use Amazon S3's multipart upload API, then incomplete multipart uploads might leave behind the uploaded parts.

Here are some additional best practices when cleaning up incomplete multipart uploads:

• To avoid storage charges for the uploaded parts that are left behind from an incomplete multipart upload, create a lifecycle policy. The lifecycle policy can be used to clean up incomplete multipart uploads after a certain number of days.
• Use Amazon S3 API calls to list your multipart uploads. Then, abort any multipart uploads that failed, or that you want to cancel.
• Use S3 Storage Lens to determine the number and size of incomplete multipart uploads for a specific bucket or all buckets in your account.

Additionally, you can leverage Amazon S3 Storage Lens to identify which S3 buckets contain multipart uploads. With S3 Storage Lens, you can preview how much data exists for the incomplete multipart uploads.

Delete previous versions of objects that you don't need

Warning: Be careful when deleting any objects or object versions. Amazon S3 can't restore any data that you delete from your buckets.

If you have versioning enabled on your bucket, then each object can have multiple versions. Each object version contributes to your storage costs.

To review your object versions, use the Amazon S3 console or run the ListObjectVersions API call. Then, delete any object versions that you don't need.

Note: In a version-enabled bucket, you must specify the object version ID to permanently delete an object version. If you run a simple delete request on an object in a version-enabled bucket, then Amazon S3 adds a delete marker to the object. The delete marker becomes the current version of the object, and the actual object becomes the previous version.However, the object and its versions are not actually erased.

You can also use the following:

• A lifecycle rule to manage the current and non-current versions of your objects. However, you must carefully review the parameters for your lifecycle rule, because permanent deletions in Amazon S3 are irreversible.
• S3 Storage Lens to determine the number of current versions, previous versions, and delete markers for specific buckets or all buckets in your account.

Review your storage-class transition costs

It's common to transition objects into other storage classes to reduce any ongoing storage costs. However, it's also important to know that there can be costs associated with transitioning objects from one storage class to another.

For example, you're charged $0.01 for every 1,000 lifecycle transition requests when objects transition from S3 Standard to the S3 Standard-IA storage class. You're charged $0.05 for every 1,000 lifecycle transition requests for objects transitioning from any storage class to the Amazon Simple Storage Service Glacier storage class. For more information and pricing details, see Amazon S3 pricing.

If you set up a lifecycle configuration rule that transitions an entire bucket's objects to another storage class, then expect an increase in S3 charges. To determine if an increase in your charges is because of a lifecycle rule, review your AWS CloudTrail event history. You can also review your Amazon S3 server access logs for PutBucketLifecycleConfiguration calls to identify when the rule was applied.

Additionally, you can use S3 Storage Class Analysis to determine how much data has been retrieved from the different storage classes.

Note: S3 Storage Class Analysis must be configured on a per bucket basis.

Review your data retrieval costs

Objects stored in the S3 Standard-IA, S3 One Zone-IA, S3 Glacier, or S3 Glacier Deep Archive storage classes have costs associated with data retrievals.

For example, if you're retrieving 15 million objects from the S3 Glacier storage class, then pay attention to the retrieval options. The total cost of retrieval will be based on the bulk retrieval option that you choose.

Bulk retrieval:

• The retrieval request is 15,000,000 / 1000 * $0.025 = $375.
• The retrieval is 100,000 * $0.0025 = $250.
• The total cost is $625.

Standard retrieval:

• The retrieval request is 15,000,000 / 1000 * $0.05 = $750.
• The retrieval is 100,000 * $0.01 = $1,000.
• The total cost is $1750.

Expedited retrieval:

• The retrieval request is 15,000,000 / 1000 * $10 = $150,000.
• The retrieval is 100,000 * $0.03 = $3,000.
• The total cost is $153,000.

Tip: To help you monitor the different storage requests and data usage, you can also use the S3 Storage Lens dashboard.

Track the requests made to your bucket

You can monitor requests to your bucket in one or more of the following ways:

• You can enable server access logging. For more information on how to review access logs, see Amazon S3 Server Access Log Format.
• You can enable object-level logging using AWS CloudTrail.
• You can enable Amazon CloudWatch metrics for Amazon S3 requests. Metrics such as AllRequests and BytesDownloaded can help you monitor the requests made to your bucket.

After you understand the requests that are made to your bucket, you can take measures to reduce your costs from requests. For example, you can prevent unauthorized access or limit public access to your bucket using bucket policies or AWS Identity and Access Management (IAM) policies. Or, to reduce your costs from data transfer OUT charges, you can access your bucket using an Amazon Elastic Compute Cloud (Amazon EC2) instance. The Amazon EC2 instance must be in an Amazon Virtual Private Cloud (Amazon VPC) within the same AWS Region as your bucket.

Check for changes in the size of your bucket

To isolate storage costs, you can compare the current size of your bucket and with the size of your bucket from a previous month. You can review your bucket's storage size in the Amazon S3 console:

1.    View your bucket from the Amazon S3 console.

2.    Choose the Management tab.

3.    Choose Metrics.

You can also review your bucket's storage size using the AWS CLI or you can monitor your storage metrics with Amazon CloudWatch. Or, to monitor bucket sizing changes on a daily, weekly and monthly basis, you can also use S3 Storage Lens.

Review the cost of individual buckets

To identify which particular bucket is resulting in higher costs, activate the cost allocation tags on your buckets. Then, review your buckets using the AWS Cost Explorer. For more information about this approach, see How do I find the cost of my Amazon S3 buckets?

Understand how your usage relates to your charges

Your usage report records storage usage in byte-hours, and your billing report records storage usage in GB-months. To understand how your storage usage relates to your storage charges, you must convert byte-hours into GB-months. For an example of this conversion, see Understanding your AWS billing and usage reports for Amazon S3.

You can download your Amazon S3 usage report from the Billing & Cost Management dashboard. Or, to obtain detailed storage and billing metrics, create an S3 Storage Lens dashboard for all the buckets in your account.

---