Greetings:
We're currently operating a Transfer Family server for the usual SFTP endpoint scenarios. As our customers upload their files to Transfer Family, we use a custom workflow to trigger a Lambda which starts the 'processing' of that file. This all currently works just fine, we're happy with it.
One edge case we're running into is if a customer uploads a file and then quickly renames the file via their SFTP client. In most cases, these rename operations happen within ~10 seconds of the original file upload. As mentioned above, the Lambda that kicks off the processing of this file still refers to the pre-rename file path, which now due to the rename, isn't a valid path anymore in S3.
We've done some testing and found that the custom Transfer Family workflow that we have isn't triggered at all for the 'file rename' event, meaning we never see a 'rename' event in the custom workflow which is our only way to trigger file processing currently. From the customer's point of view, their uploaded file is never processed at all. From a developer's point of view, Transfer Family has allowed the customer to do something that we can't action against because we're never notified about it using existing workflows. Of course, a solution could exist and we haven't found it, I would be excited to be wrong. :)
The only record that we have that even registers the file rename event is via the CloudWatch logs that we have set up to receive from Transfer Family. The event data looks like this in those logs
23423485a3fead4834763451c0ee8c02a.ed234gfds3d2373 RENAME Path=/foobar/myfile.csv NewPath=/foobar/myfile_renamed.csv
My question is -- is there a product solution to this? Are file renames a use case in Transfer Family? After researching, it seems like no. I might be able to add something to parse CloudWatch logs to look for RENAME in log entries above, but that seems .. hacky. Has anyone created a suitable solution to this problem recently? Thanks for reading, appreciate your time.
AWS OFFICIALUpdated 3 years ago