Why is there a discrepancy in storage and size metrics between the Amazon S3 console, CloudWatch, and S3 Storage Lens?
I'm seeing a discrepancy between the following: “Calculate total size” number in the Amazon Simple Storage Service (Amazon S3) console, daily storage metric for buckets in Amazon CloudWatch, and the "Total storage" metric in S3 Storage Lens.
When you see this discrepancy, check whether the following is true:
- You turned on object versioning.
Note: The object versioning feature in Amazon S3 retains multiple versions of an object in your bucket. By default, Amazon S3 object versioning is turned off on buckets, and you must explicitly turn this feature on.
- Incomplete multipart uploads exist.
Note: Incomplete multipart uploads aren't visible in the Amazon S3 console, but are calculated as part of the storage in CloudWatch and Storage Lens metrics.
To identify the cause of the reporting discrepancy, check whether you turned on object versioning. Also, look for any multipart uploads in your bucket. These two factors might result in an increased value of the calculated bucket size in CloudWatch and the Total storage metric S3 Storage Lens. These factors aren't used when calculating the total size of the object in the S3 console. For more information, see Amazon S3 daily storage metrics for buckets in CloudWatch.
Tip: If you have incomplete multipart uploads in Amazon S3, then consider creating a lifecycle configuration rule. This lifecycle configuration rule automatically cleans up any incomplete parts, lowering the cost of data storage. Note that lifecycle rules operate asynchronously, so there might be a delay with the operation. However, as soon as the objects are marked for deletion, you are no longer billed for storage (even if the object isn't removed yet).
Note that the Amazon S3 monitoring metrics are recorded once per day. Therefore, these metrics might not display the most updated information. However, CloudWatch monitors your AWS resources and applications in real time. Also, S3 console and Storage Lens use base 2 conversion (/1024) to report storage metrics, and CloudWatch by default uses base 10 conversion (/1000).
Daily storage metrics in CloudWatch
In CloudWatch, the BucketSizeBytes metric captures all Amazon S3 and Amazon S3 Glacier storage types, object versions, and any incomplete multipart uploads. This value is calculated by summing up all object sizes, metadata in your bucket (both current and noncurrent objects), and any incomplete multipart upload sizes. For example, the BucketSizeBytes metric calculates the amount of data (in bytes) that's stored in an Amazon S3 bucket in all the following object storage classes:
- S3 Standard
- S3 Intelligent-Tiering
- S3 Standard-IA
- S3 One Zone-IA
- S3 Reduced Redundancy Storage
- S3 Glacier Deep Archive
- S3 Glacier Flexible Retrieval
- S3 Glacier Instant Retrieval
Additionally, the NumberOfObjects metric in CloudWatch contains the total number of objects that are stored in a bucket for all storage classes. This value counts all objects in the bucket (both current and noncurrent), along with the total number of parts for any incomplete multipart uploads. The NumberOfObjects metric also calculates the total number of objects for all versions of objects in your bucket. For example, if you have two versions of the same object, then the two versions are counted as two separate objects. For more information, see Metrics and dimensions.
"Total storage" metric in S3 Storage Lens dashboard
You can see the Total storage metric in the S3 Storage Lens dashboard under the Bucket tab. This metric captures the total storage, including incomplete multipart uploads, object metadata, and delete markers. To see the exact composition of space that's occupied by noncurrent versions, incomplete multipart uploads, or delete markers, check individual metrics, such as the following:
- Current version bytes
- Noncurrent version bytes
- Incomplete multipart upload bytes
These metrics are available under AWS Free Tier.
Also, the Object count metric includes information on current and noncurrent versions, delete markers, and incomplete multipart upload object counts. For granular level information, check metrics, such as the following:
- Current version object count
- Noncurrent version object count
- Delete marker object count
- Incomplete multipart upload object count.
For more information, see Amazon S3 Storage Lens metrics glossary.
"Calculate total size" in Amazon S3 console
To calculate the size of your bucket from the Amazon S3 console, you can use the Calculate total size action. Amazon S3 then calculates your bucket's storage size. However, note that multipart uploads, and previous or noncurrent versions aren't calculated in the total bucket size. Amazon S3 calculates only the total number of objects for the current or newest version of each object that is stored in the bucket. For example, if there are two versions of an object in your bucket, then Amazon S3's storage calculator counts them as only one object. As a result, the number that is calculated in the Amazon S3 console is smaller than the one reported by CloudWatch.
Incomplete multipart uploads
To check the amount of size that's occupied by incomplete multipart uploads, check the Incomplete multipart upload bytes metric in the S3 Storage Lens dashboard under the Bucket tab.
To review the list of incomplete multipart uploads, run the list-multipart-uploads command:
aws s3api list-multipart-uploads --bucket <bucket-example>
Then, list all the objects in the multipart upload, using the list-parts command and your UploadId value:
aws s3api list-parts --bucket <bucket-example> --key large_test_file --upload-id <examplevalue>
Creating a lifecycle rule
To automatically delete multipart uploads, you can create a lifecycle configuration rule:
1. Open the Amazon S3 console.
2. Choose the Management tab.
3. Choose Create new policy.
4. Add the name of the policy.
5. Choose Select - Delete expired delete markers or incomplete multipart uploads.
6. (Optional) If your bucket isn't versioned, then choose Delete incomplete multipart uploads.
To review and audit your Amazon S3 bucket for different versions of objects, use the Amazon S3 inventory list. An Amazon S3 inventory list file contains a list of the objects in the source bucket and metadata for each object. The inventory list file capture metadata information such as bucket name, object size, storage class, and version ID. To calculate the amount of size that's occupied by current and noncurrent versions, check the S3 Storage Lens dashboard metric Current-version bytes and Non-current version bytes, respectively.
Example 8: Lifecycle configuration to abandon multipart uploads
Expiring objectsAmazon S3 daily storage metrics for buckets in CloudWatch
- rePost-User-1431093lg...asked 5 months agolg...
- landdoggerlg...asked 3 years agolg...
- ISanVlg...asked 2 years agolg...
- Accepted AnswerManuellg...asked 2 months agolg...
- Accepted AnswerZerdalg...asked 5 months agolg...
- AWS OFFICIALUpdated 15 days ago
- Why is there a discrepancy between the sum of some historical metrics and another metric in Amazon Connect?AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated 7 months ago
- AWS OFFICIALUpdated a year ago
- EXPERTpublished 4 months agolg...