S3 File Gateway file limit per SMB share

0

I heard second hand from someone working with AWS support that there is a 20 million file limit per share when using S3 File Gateway. Can someone confirm is this limit is true as I do not see any AWS documentation stating a file count per share limit.

Thanks.

asked 2 months ago297 views
3 Answers
2

Hii,

There isn't a documented file limit per SMB share with S3 File Gateway. While some suggest a soft limit of 20 million, it's best to plan for large file counts. Consider creating multiple shares or using EFS/FSx for massive amounts of small files.

  • Large file count? Plan for multiple SMB shares (up to 50 per gateway) or explore EFS/FSx for very high numbers of small files.

https://aws.amazon.com/storagegateway/file/s3/

profile picture
EXPERT
answered 2 months ago
  • Unfortunately we are contemplating a migration of over 200TB from an on premises storage solution to S3 with a requirement that existing SMB clients are not impacted. There is no easy way to partition the existing file systems containing 100+ million files, so I am trying to understand whether there is a hard limit to the file count. It was suggested that Storage Gateway caches the entire share catalog in memory on the VM, wherein the hard 20 million file limit derives.

1

Migrating a large dataset with over 100 million files and 200TB to AWS S3 using S3 File Gateway is a significant task. While the official AWS documentation does not specify a hard file limit per SMB share, there are practical considerations related to performance and resource utilization, such as memory usage for caching metadata.

Here's a concise summary:

  • No Official Limit: AWS documentation does not specify a hard file count limit for S3 File Gateway SMB shares.
  • Practical Limitations: The suggestion that S3 File Gateway caches the entire share catalog in memory is plausible. This means that a very high file count, like 20 million or more, could be constrained by the available memory and resources of the gateway VM.
  • Performance Impact: Managing very large numbers of files can impact performance, particularly due to metadata handling and resource constraints.
profile picture
EXPERT
answered 2 months ago
0

There is no hard limit on how many files a S3 File Gateway file share can hold as the shares are backed by s3(it’s virtually unlimited).

Gateway maintains two different types of cache. One is the data cache and other is the metadata cache. While frequently accessed data is cached in the location cache disk attached to the gateway (supports up to 64TiB) metadata cache is a cache used to store local copy of the metadata for each file (file name, size, attributes, ACL, etc.) which is stored in the root disk. S3 File Gateway can be configured for 5M, 10M, or 20M metadata cache entries. Depending on the configured gateway capacity, gateway can hold up to the configured cache entries in the memory for faster file listing. Gateway dynamically evicts and fetches entries if the requested data is not already cached. And this can add latency to reads if the gateway hit the configured metadata cache limit.

Performance impact depending on the use case:

  1. There can be added latency to the read requests if the application requests the dataset size which is beyond the configurable metadata entry limit. Or if the application reads are random.
  2. For write intensive work loads ( storing backups), for optimal performance make sure you deploy your storage gateway closer to your client systems on which you will mounting the file share and have enough network bandwidth between your storage gateway and AWS Endpoints for both data & control plane communication. Here is guidance on ways to optimize gateway performance.
Harshi
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions