List, Delete operations with SFTP Connectors

0

I'm setting up an SFTP connector to sweep files from a client's server. The workflow is as follows, get the list of files available to download, trigger the download using the SFTP connector, wait for the connector to download the file, trigger the delete operation on the remote server. To make this work, I'm using Lambda functions for each of the operations and having to make individual SFTP connections for each file download and delete. In my use case, I have close to 200 files in the server and making 200 connections to download and 200 connections to delete the files seems like a lot.

Is there a way to create a workflow to trigger the connector (using EventBridge or Lambda trigger) with the external SFTP sweep path and delete permissions flag and for the service to manage sweeping subdirectories, download all available files and then delete them if the permission is enabled? This would be great if it is offered as a managed service in AWS. One other thing to manage would be to handle files in transit to the remote server, handling currently in Lambda.

2 個答案
1
已接受的答案

Hi! You can design a workflow such that you use SFTP connectors to List files (StartDirectoryListing operation). Once file list is generated, connector emits an event to EventBridge, and you can use this event to trigger a lambda function that parses the file list and request each file path for transfer using the SFTP connectors' StartFileTransfer operation. Connectors also emit an event to EventBridge after eachfile is transferred, so that event can be used to trigger a custom lambda to delete files from remote server. Currently, SFTP connectors provide operations to List and to Transfer files, but not to delete files.

AWS
已回答 6 天前
  • Exactly, the delete files functionality is key here and it is unavailable. Will wait until that is enabled on the connector service.

-1

Consolidate your file operations into batches using a single SFTP connection for downloading and deleting multiple files at once, reducing the overhead of establishing numerous connections. This can significantly improve efficiency and reduce the overall processing time.

profile picture
專家
已回答 1 個月前
  • SFTP connectors (service offered by AWS) doesn't allow batch file operations or delete capabilities. I've reviewed the documentation available.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南