Transfer Family vs Lambda Function for file transfer from SFTP server

Question

Hello Community,
I was trying to narrow down to use one of the options to transfer files from the SFTP server to the S3 bucket, so as to help my Glue jobs because AWS Glue doesn’t support data loads from other cloud applications, File Storages. So, here I am trying to choose one, either AWS transfer Family or AWS lambda function that can connect to the remote server, and move them into the S3 bucket/folder which becomes the source of my integrations. 
I greatly appreciate it if you could share some insights into this scenario, and the advantages, and drawbacks of choosing one over the other. Any bottlenecks that you guys faced in using either of these services before for file transfer? Which is more cost-effective, suppose we say we have gigabytes of data(files).

Thank you.

Best,
Tharun

Nuno_Q · Answer

Hi,

Lambda would be better suited for post-delivery data processing operations AFTER the files were fully delivered. Lambda would play no role in the direct SFTP file transfer.

Depending on how many files would be transferred or it's size the Lambda function could reach it's limits https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

One of the limits to keep in mind is the Lambda function timeout of 900 seconds (15 min)

Benefits of using AWS Transfer Family are detailed here: https://aws.amazon.com/aws-transfer-family/mft

Features of AWS Transfer Family are detailed here: https://aws.amazon.com/aws-transfer-family/features

Several blog posts that detail how to setup AWS Transfer Family can be setup and deployed to protect your File Transfer workflows:

https://aws.amazon.com/blogs/storage/configuring-your-ftps-server-behind-a-firewall-or-nat-with-aws-transfer-family/

https://aws.amazon.com/blogs/storage/secure-your-aws-transfer-family-sftp-and-ftps-vpc-public-endpoints/

Indranil Banerjee AWS · Answer

What Nuno mentioned is correct. Lambda is not a good option for moving files from on-prem to AWS. I would like to make you aware of some other options as well for moving files from on-prem to AWS S3.

If you want to move files from on-prem to S3 on a schedule, you can also look at AWS DataSync - https://aws.amazon.com/datasync/

There are also some good blogs on AWS DataSync - https://aws.amazon.com/blogs/storage/tag/aws-datasync/

If you want the ability to also access files on S3 for your on-prem users, you can also look at File Gateway - https://aws.amazon.com/storagegateway/file/

There are blogs on AWS File Gateway - https://aws.amazon.com/blogs/storage/tag/aws-file-gateway/

You can look at the AWS Cost Calculators for each of these services to estimate the costs based on your usage patterns.

Transfer Family vs Lambda Function for file transfer from SFTP server

Relevant content