Some questions about AWS Storage Gateway

Hello, Thank you for reaching out. To answer your questions...

1. Yes, that is correct. The purpose of refresh cache feature is to refresh the gateway's metadata cache inventory with any changes that are made outside the gateway or directly in the S3 bucket. If the changes are made on the File Share via SMB or NFS client then gateway will have its metadata cache updated.

2. When a file is modified in the S3 File Gateway by an NFS or SMB client after it has been uploaded to Amazon S3, the S3 File Gateway uploads the new or modified data instead of uploading the whole file. This process primarily uses CPU resources and not necessarily bandwidth. The File Gateway compares the blocks of the new file version with the existing blocks in storage locally. Increased bandwidth usage occurs when the comparison reveals that many or all blocks have changed and need to be uploaded. In cases where 100% of the file blocks are different (like overwriting encrypted or compressed files), the entire file needs to be uploaded, resulting in maximum bandwidth usage. For files with minimal changes, only the changed blocks are uploaded, which reduces bandwidth usage compared to uploading the entire file. When uploading completely new files, there's no comparison needed, but full bandwidth is used to upload the entire file.

That said, the service optimizes data transfer between the gateway and AWS using multipart parallel uploads or byte-range downloads, to better use the available bandwidth. Local cache is maintained to provide low latency access to the recently accessed data and reduce data egress charges.

There could be multiple other factors contributing to slow upload performance. You can monitor the Client (SMB/NFS Client to Storage Gateway local cache) and cloud (From local cache to AWS S3) upload speeds using the CloudWatch Metrics during the process of file upload and modification to see the BW and CPU utilization. Also, you can monitor the IOWaitPercent metric in CloudWatch, which reports the percentage of time that the CPU is waiting for a response from the local disk.. A value higher than 10% typically indicates a bottleneck in the underlying disks and can be a result of slower disks. Which could lead to slowness.

Thank you for your questions about AWS Storage Gateway. I'll address each of your points:

1. Refresh Cache operation: Yes, your understanding is correct. The Refresh Cache operation primarily lists objects in S3, retrieves metadata of those objects, and updates the local file gateway cache with this metadata. It doesn't download the actual file contents, just the metadata.

2. Updating files with the same name: When you update files with the same name in the file share, Storage Gateway will detect the changes and initiate an upload to S3. This process is typically triggered quickly, but the exact timing can depend on factors like file size and network conditions. The sync process is designed to be responsive to file changes.

3. Refresh Cache and file updates: Refresh Cache doesn't directly trigger file uploads to S3. Its primary function is to update the local cache with the latest metadata from S3. If there are local changes that haven't been uploaded yet, the Refresh Cache operation won't initiate those uploads.

4. File upload order: Storage Gateway doesn't have a strict "first in, first out" mechanism for file uploads. The order of uploads can depend on various factors, including file size and system load. Generally, smaller files might upload faster than larger ones, but there's no guaranteed order for simultaneous uploads.

5. Monitoring and prioritizing uploads: Unfortunately, Storage Gateway doesn't provide a built-in way to see which specific file is currently uploading or to change the upload order of files. The service manages the upload queue internally. If you need to prioritize a specific file, you might consider uploading it separately or at a different time.

6. Automated cache refresh: "Accessing a directory" typically means any operation that reads the directory contents, such as listing files or opening the directory in a file explorer. Simply putting a file into the file share doesn't usually trigger a cache refresh. The TTL setting determines how often the gateway checks S3 for changes, regardless of local access.

Remember that while Storage Gateway provides a seamless interface between your on-premises environment and S3, some of its internal operations are abstracted away from direct user control for simplicity and efficiency.
Sources
Storage Gateway - AWS Prescriptive Guidance
AWS Storage Gateway Features | Amazon Web Services
Managing gateway updates - AWS Storage Gateway
Monitoring your S3 File Gateway - AWS Storage Gateway

To elaborate on the automated answer:

1. Imagine someone writes to the S3 bucket directly. The "refresh cache" allows the new meta data to be cached, as the file gateway, to that point is only aware as to what it has been handling. So, this allows for the file gateway cache to be aware, as new files could land via a path where the gateway is normally unaware.
2. The file gateway, for efficiency, does a block comparison when it does an upload. So, if a minimal amount of a file has changed, it only has to upload that difference. In your case, you are overwriting 100% of the blocks - so it is doing that comparison where in a new file it would not. (if you are overwriting encrypted files and possibly with compressed files - you would also see 100% file change). So, less CPU is being used - however, I would assume in most cases, your bandwidth is going to be the bottleneck. And your blocks that need uploading end up in the queue.

Dear expert:

Thank you very much for the information provided.

According to the explanation and the automated answer, there are still some points that I would like to confirm.

1. According to the explanation and the automated answer, refresh cache could aware file gateway that some objects were changed(create/update/delete) direct in S3 bucket.

   • If there are some files haven't been upload to S3, refresh cache can not help with this situation. (Because file gateway is aware of files change local in the file share, no matter the files were create or update, but not objects change in S3)

2. We did facing files uploading problems, and it is now seems to caused by the internet, bandwidth, network speed or throughput.(We are still in troubleshooting to find out the root cause)

   • Since file gateway will do block comparison when we update files, could I consider that, delete the files that are already upload to S3, and then create new files, could save efforts of the gateway?
   • Could delete the files first and then create the new one with the same file name, let the file gateway recognize this as an create action, and save the process of block comparison?
   • It seems that block comparison need more bandwidth rather than CPU, if the file gateway does lots of block comparison for a period of time, could this use case cause some negative effect to file gateway? (For example, some negative Influence on efficiency)
   • Could you tell us that why block comparison need more bandwidth, but not need more CPU ? (Since we used to thought that compare files might use high CPU)