List, Delete operations with SFTP Connectors

0

I'm setting up an SFTP connector to sweep files from a client's server. The workflow is as follows, get the list of files available to download, trigger the download using the SFTP connector, wait for the connector to download the file, trigger the delete operation on the remote server. To make this work, I'm using Lambda functions for each of the operations and having to make individual SFTP connections for each file download and delete. In my use case, I have close to 200 files in the server and making 200 connections to download and 200 connections to delete the files seems like a lot.

Is there a way to create a workflow to trigger the connector (using EventBridge or Lambda trigger) with the external SFTP sweep path and delete permissions flag and for the service to manage sweeping subdirectories, download all available files and then delete them if the permission is enabled? This would be great if it is offered as a managed service in AWS. One other thing to manage would be to handle files in transit to the remote server, handling currently in Lambda.

2 回答
1
已接受的回答

Hi! You can design a workflow such that you use SFTP connectors to List files (StartDirectoryListing operation). Once file list is generated, connector emits an event to EventBridge, and you can use this event to trigger a lambda function that parses the file list and request each file path for transfer using the SFTP connectors' StartFileTransfer operation. Connectors also emit an event to EventBridge after eachfile is transferred, so that event can be used to trigger a custom lambda to delete files from remote server. Currently, SFTP connectors provide operations to List and to Transfer files, but not to delete files.

AWS
已回答 6 天前
  • Exactly, the delete files functionality is key here and it is unavailable. Will wait until that is enabled on the connector service.

-1

Consolidate your file operations into batches using a single SFTP connection for downloading and deleting multiple files at once, reducing the overhead of establishing numerous connections. This can significantly improve efficiency and reduce the overall processing time.

profile picture
专家
已回答 1 个月前
  • SFTP connectors (service offered by AWS) doesn't allow batch file operations or delete capabilities. I've reviewed the documentation available.

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则