Storage Gateway Rename Operation

Question

Hi. I'm using Storage Gateway S3 File Gateway to sync file from onprem to S3. When checking the log, I see SGW will first create a file with some random extension name like this:

```
{
    "sourceAddress": "xxx.xxx.xxx.xxx",
    "clientUid": "0",
    "source": "share-xxx",
    "type": "FileSystemAudit",
    "version": "1.0",
    "objectType": "File",
    "bucket": "bucket-name",
    "clientGid": "0",
    "objectName": "/.test.TXT.JsumvE",
    "shareName": "/file-share-name",
    "operation": "WriteData",
    "gateway": "sgw-xxx",
    "timestamp": "xxxxxxxxx",
    "status": "Success"
}
```
After finishing data and attributes, I see another log like this:

```
{
    "sourceAddress": "xxx.xxx.xxx.xxx",
    "clientUid": "0",
    "source": "share-xxx",
    "type": "FileSystemAudit",
    "version": "1.0",
    "objectType": "File",
    "bucket": "bucket-name",
    "clientGid": "0",
    "objectName": "/.test.TXT.JsumvE",
    "shareName": "/file-share-name",
    "operation": "Rename",
    "newObjectName": "/test.TXT",
    "gateway": "sgw-xxx",
    "timestamp": "xxxxxxxxx",
    "status": "Success"
}
```
With other file, I no longer see the "temporary file name" in S3 after the Rename operation, but with this file, the temporary file with extension .JsumvE is still there with the renamed file. I'm guessing this is a cache file waiting for change or something like that??? Is this the default behaviour of Storage Gateway or is there any error during the syncing operation? Thanks in advance

Accepted Answer

From the extension of temporary file, it looks like "rsync" tool was used to sync files from onprem to File Gateway. "rsync" always creates temporary files first, copies the data into it and then renames the temporary file to final name. The Audit logs also point towards the same behavior. File Gateway syncs the request made by the rsync tool into S3 and does not create any temporary files on its own. You might see both files (temp and final) in S3 for a brief period but eventually the file gateway will upload all changes. You can monitor the "CachePercentDirty" metric and ensure it remains at 0 consistently to indicate completion of upload, or setup CloudWatch Events to get notified when file uploads are done.

https://docs.aws.amazon.com/filegateway/latest/files3/file-gateway-concepts.html https://docs.aws.amazon.com/filegateway/latest/files3/monitoring-file-gateway.html#get-notification

https://docs.aws.amazon.com/filegateway/latest/files3/file-gateway-concepts.html
https://docs.aws.amazon.com/filegateway/latest/files3/monitoring-file-gateway.html#get-notification

Storage Gateway Rename Operation

Relevant content