S3 File Gateway file limit per SMB share

Hii,

There isn't a documented file limit per SMB share with S3 File Gateway. While some suggest a soft limit of 20 million, it's best to plan for large file counts. Consider creating multiple shares or using EFS/FSx for massive amounts of small files.

• Large file count? Plan for multiple SMB shares (up to 50 per gateway) or explore EFS/FSx for very high numbers of small files.

https://aws.amazon.com/storagegateway/file/s3/

Migrating a large dataset with over 100 million files and 200TB to AWS S3 using S3 File Gateway is a significant task. While the official AWS documentation does not specify a hard file limit per SMB share, there are practical considerations related to performance and resource utilization, such as memory usage for caching metadata.

Here's a concise summary:

• No Official Limit: AWS documentation does not specify a hard file count limit for S3 File Gateway SMB shares.
• Practical Limitations: The suggestion that S3 File Gateway caches the entire share catalog in memory is plausible. This means that a very high file count, like 20 million or more, could be constrained by the available memory and resources of the gateway VM.
• Performance Impact: Managing very large numbers of files can impact performance, particularly due to metadata handling and resource constraints.

There is no hard limit on how many files a S3 File Gateway file share can hold as the shares are backed by s3(it’s virtually unlimited).

Gateway maintains two different types of cache. One is the data cache and other is the metadata cache. While frequently accessed data is cached in the location cache disk attached to the gateway (supports up to 64TiB) metadata cache is a cache used to store local copy of the metadata for each file (file name, size, attributes, ACL, etc.) which is stored in the root disk. S3 File Gateway can be configured for 5M, 10M, or 20M metadata cache entries. Depending on the configured gateway capacity, gateway can hold up to the configured cache entries in the memory for faster file listing. Gateway dynamically evicts and fetches entries if the requested data is not already cached. And this can add latency to reads if the gateway hit the configured metadata cache limit.

Performance impact depending on the use case:

1. There can be added latency to the read requests if the application requests the dataset size which is beyond the configurable metadata entry limit. Or if the application reads are random.
2. For write intensive work loads ( storing backups), for optimal performance make sure you deploy your storage gateway closer to your client systems on which you will mounting the file share and have enough network bandwidth between your storage gateway and AWS Endpoints for both data & control plane communication. Here is guidance on ways to optimize gateway performance.