List, Delete operations with SFTP Connectors

0

I'm setting up an SFTP connector to sweep files from a client's server. The workflow is as follows, get the list of files available to download, trigger the download using the SFTP connector, wait for the connector to download the file, trigger the delete operation on the remote server. To make this work, I'm using Lambda functions for each of the operations and having to make individual SFTP connections for each file download and delete. In my use case, I have close to 200 files in the server and making 200 connections to download and 200 connections to delete the files seems like a lot.

Is there a way to create a workflow to trigger the connector (using EventBridge or Lambda trigger) with the external SFTP sweep path and delete permissions flag and for the service to manage sweeping subdirectories, download all available files and then delete them if the permission is enabled? This would be great if it is offered as a managed service in AWS. One other thing to manage would be to handle files in transit to the remote server, handling currently in Lambda.

1 Answer
-1

Consolidate your file operations into batches using a single SFTP connection for downloading and deleting multiple files at once, reducing the overhead of establishing numerous connections. This can significantly improve efficiency and reduce the overall processing time.

profile picture
EXPERT
answered 18 days ago
  • SFTP connectors (service offered by AWS) doesn't allow batch file operations or delete capabilities. I've reviewed the documentation available.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions