How do I reduce unexpected high costs for Amazon S3?

Short description

To identify what type of Amazon S3 charges you incur, review your current billing and usage for Amazon S3. You can also use the AWS usage reports to view the S3 buckets that incur charges.

Based on the type of usage, use the following methods to help you understand your billing:

• Use Amazon S3 Storage Lens to view usage and activity across your S3 storage. The default dashboard updates daily with the storage metrics for all S3 buckets in your AWS account.
• Use server-access logging or AWS CloudTrail data events to monitor requests to your S3 buckets.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

To reduce your Amazon S3 billing costs, take the following actions.

Delete incomplete multipart uploads

After you initiate a multipart upload, Amazon S3 retains all the uploaded parts of the file until you complete or end the multipart upload. If you don't complete the multipart upload, then Amazon S3 continues to store the uploaded parts, and you're charged for the storage.

When you run the abort-multipart-upload AWS CLI command to end the multipart upload, the command deletes the incomplete uploaded parts. However, other tools might keep the incomplete uploaded parts when you end the multipart upload.

To delete incomplete multipart uploads, use the following best practices:

• Create a lifecycle policy to delete incomplete multipart uploads after a specified number of days.
• Use the ListMultipartUploads Amazon S3 API call to list your multipart uploads. Then, use the AbortMultipartUpload to end multipart uploads that failed or that you want to cancel.
• Use Amazon S3 Storage Lens to determine the number and size of incomplete multipart uploads for buckets in your account.
• Use Amazon S3 Storage Lens to identify S3 buckets that contain multipart uploads.

Delete previous versions of objects

Important: Amazon S3 can't restore data that you delete from your buckets.

If you activated the S3 Versioning feature on your bucket, then each object can have multiple versions that contribute to your storage costs.

To review your object versions, use the Amazon S3 console or run the ListObjectVersions API call. Then, delete object versions that you don't need.

In a version-activated bucket, you must specify the object version ID to permanently delete an object version. If you run a delete request on the object, then Amazon S3 adds a delete marker to the object. The delete marker becomes the current version of the object, and the object becomes the previous version. However, Amazon S3 doesn't delete the object and its versions.

To manage object versions, take the following actions:

• Create a lifecycle rule.
  Important: You can't reverse permanent deletions in Amazon S3. Make sure to review the parameters for your lifecycle rule.
• Use Amazon S3 Storage Lens metrics to determine the number of current and previous versions and delete markers for buckets in your account.

Review your storage-class change costs

You might incur costs when you change objects from one storage class to another.

Also, AWS charges for 40 KB of additional metadata for each archived object that you store in S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive. You're charged 8 KB at the S3 Standard rate and 32 KB at the S3 Glacier Flexible Retrieval or S3 Deep Archive rate.

For more information, see Transitioning objects using Amazon S3 Lifecycle.

It's a best practice to aggregate multiple small objects into a few large objects to reduce overhead costs.

When you set up a lifecycle configuration rule that transitions an entire bucket's objects to another storage class, you incur Amazon S3 charges. To determine whether a lifecycle rule caused a cost increase, review your AWS CloudTrail event history and Amazon S3 server access logs for PutBucketLifecycleConfiguration calls. Also, check the logs for S3 Lifecycle actions.

To optimize storage costs, it's a best practice to use the S3 Intelligent-Tiering storage class that automatically moves data based on your access requirements.

You can also use S3 storage class analysis to determine how much data is in each storage class.

Note: You must configure S3 storage class analysis for each bucket.

Review your data retrieval costs

Objects that you store in the following storage classes have data retrieval costs:

• S3 Standard-IA
• S3 One Zone-IA
• S3 Glacier Flexible Retrieval
• S3 Glacier Deep Archive storage classes

For example, you retrieve 15 million objects from the S3 Glacier Flexible Retrieval storage class. The total cost of retrieval is based on the bulk retrieval option that you choose.

The following example costs are for a standard retrieval:

• The retrieval request is 15,000,000 / 1,000 * $0.05 = $750.
• The retrieval is 100,000 * $0.01 = $1,000.
• The total cost is $1750.

The following example costs are for an expedited retrieval:

• The retrieval request is 15,000,000 / 1,000 * $10 = $150,000.
• The retrieval is 100,000 * $0.03 = $3,000.
• The total cost is $153,000.

There's no charge for bulk retrievals of objects that you store in S3 Glacier Flexible Retrieval. To help you monitor the different storage requests and data usage, use the S3 Storage Lens dashboard.

Monitor the requests to your bucket

Use the following methods to monitor requests to your bucket and review incurred charges from a high number of requests:

• AWS doesn't charge you to turn on server access logging. However, you incur the usual standard storage charges for the log files that the feature delivers to you.
• You incur charges to use AWS CloudTrail object-level logging.
• You incur charges to use Amazon CloudWatch metrics.

To reduce your costs from data transfer OUT charges, you can use an Amazon Elastic Compute Cloud (Amazon EC2) instance to access your bucket. The Amazon EC2 instance must be in an Amazon Virtual Private Cloud (Amazon VPC) within the same AWS Region as your bucket.

Review changes to the size of your bucket

To compare the current and previous size of your bucket, complete the following steps:

1. Open the Amazon S3 console.
2. Choose Buckets, and then select your bucket.
3. Choose the Management tab.
4. Choose Metrics.

Or, run the following ls AWS CLI command:

aws s3 ls --summarize --human-readable --recursive s3:///bucket-name

Note: Replace bucket-name with your bucket's name.

Example output:

2021-10-07 21:32:57 452 Bytes foo/bar/car/petrol
2021-10-07 21:32:57 896 Bytes foo/bar/truck/diesel
2021-10-07 21:32:57 189 Bytes foo/bar/hybrid/battery
2021-10-07 21:32:57 398 Bytes vehicles.txt

Total Objects: 4
Total Size: 2.9 MiB

 You can also use CloudWatch to monitor your storage metrics. To monitor bucket size changes on a daily, weekly, and monthly basis, use S3 Storage Lens.

Review the cost of individual buckets

To identify bucket that incurs higher costs, activate cost allocation tags on your buckets. Then, use AWS Cost Explorer to review your buckets. For more information, see How do I find the cost of my Amazon S3 buckets?

Understand how your usage relates to your charges

Your usage report records storage usage in byte-hours, and your billing report records storage usage in gigabyte-months (GB-months). To understand how your storage usage relates to your storage charges, convert byte-hours into GB-months. To view an example conversion, see How will I be charged and billed for my use of Amazon S3? on Amazon S3 FAQs.

You can download your Amazon S3 usage report from the AWS Billing and Cost Management dashboard. Or, create an S3 Storage Lens dashboard to get detailed storage and billing metrics.

Related information

Amazon S3 pricing

How do I use Amazon Athena to analyze my Amazon S3 server access logs?