S3 version disappeared if updating file in the Storage Gateway file share in ONE minute

Question

Hello everyone.

We use Storage Gateway's SMB File Share + S3 Versioning to building a backup system.
But we found S3 version missing issue  like the following:
Update File History:
```
2022-03-22 17:35:57　A
2022-03-22 17:36:51　B
2022-03-22 17:37:18　C
```
S3 Version:
```
"LastModified": "2022-03-22 17:35:58"　A
"LastModified": "2022-03-22 17:36:30"　A
"LastModified": "2022-03-22 17:37:19"　C
"LastModified": "2022-03-22 17:37:00"　C
```
the Second Update "B" to the file  is missing.
If I update file to "B" after one minute(eg: after 2022-03-22 17:37:00), B will be stored as a S3 version.

I found the comment in the aws document:

To maximize write performance, the gateway uses a write-back mechanism where data is first persisted to disk and then **asynchronously **uploaded to S3.

・Do we have to update file at least more than one minute since the last update?

・Is the "one minute" right?Would you please tell us the timing of send to S3 after storing file in the storage gateway disk.

Thanks in advance.

Ryuu

Accepted Answer

S3 File Gateway tries to optimize the number of versions uploaded to the S3 bucket in order to reduce network bandwidth and storage costs. This will sometimes result in a fewer number of objects than the number of file updates (data or metadata) that occurred on the NFS/SMB file share.

If your goal is to preserve every version of a file uploaded to a file share in the S3 bucket, then we suggest using the file upload notification feature. Please refer to the “Getting file upload notification” section in our documentation [[1][. You can also find additional details about this feature in our blog on the same topic [[2]]. This feature will enable you or your processing to wait until you receive “Storage Gateway Object Upload Event” for a file version before uploading 2nd version of that file.

[[1]] https://docs.aws.amazon.com/filegateway/latest/files3/monitoring-file-gateway.html#get-notification

[[2]] https://aws.amazon.com/blogs/storage/processing-file-upload-notifications-from-aws-storage-gateway-on-amazon-s3/

S3 version disappeared if updating file in the Storage Gateway file share in ONE minute

相关内容