Skip to content

S3 SMB share is 52TB actual data but shows as 2.7PB in CloudWatch

0

Basic SMB share S3 File Gateway - Software version - 1.27.13 US East 2

asked a year ago161 views

1 Answer
0

This discrepancy between your actual S3 data size (52TB) and what's showing in CloudWatch (2.7PB) is unusual and concerning. While there isn't a documented hard limit on the number of files an S3 File Gateway SMB share can hold (as shares are backed by S3, which is virtually unlimited), there are some important considerations that might explain what you're seeing:

  1. Metadata caching: S3 File Gateway maintains two types of cache - data cache and metadata cache. The metadata cache stores local copies of file metadata (name, size, attributes, ACLs) in the root disk. Depending on your gateway configuration, it can be set for 5M, 10M, or 20M metadata cache entries.

  2. Cache refresh issues: If changes are made directly in S3, they might not be immediately reflected in the SMB share view. This could potentially cause discrepancies in reporting. You can use the RefreshCache command to update the cache, but there might be latency involved.

  3. Performance considerations: With very large numbers of files or data volumes, there can be resource constraints related to the gateway VM's available memory, particularly for metadata handling.

  4. Storage class impact: The default storage class for objects put into an S3 bucket by the S3 File Gateway can be configured (S3_STANDARD, S3_INTELLIGENT_TIERING, S3_STANDARD_IA, or S3_ONEZONE_IA), which might affect how data is reported.

I would recommend:

  • Verifying your S3 storage size using S3 Storage Lens or the S3 console directly to confirm the actual storage used
  • Checking if the RefreshCache command has been run recently
  • Reviewing the gateway's configuration, particularly the metadata cache settings
  • Contacting AWS Support as this significant discrepancy might indicate an issue with the gateway or its reporting mechanism

This level of discrepancy (52TB vs 2.7PB) suggests there might be an underlying issue with how the data is being reported or calculated in CloudWatch.
Sources
S3 File Gateway file limit per SMB share | AWS re:Post
Changes made directly in S3 not reflected in SMB share after RefreshCache update | AWS re:Post
SMBFileShareInfo - Storage Gateway
Cutting-Edge Innovation, Yet No Simple Way to See Your S3 Storage Size? | AWS re:Post

answered a year ago

EXPERT

reviewed a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.