What configuration do I need to use to utilize Transfer Acceleration when connecting to an S3 bucket via SFTP Client (Cyberduck / WinSCP) ?

Question

I am trying to help some team mates on the other side of the world to have faster upload and download speeds to our bucket located in the US. I have enabled Transfer Acceleration on the bucket, but I do not think the SFTP clients utilize it by default. I was able to connect with one IAM user to the root of the bucket by using the endpoint <bucket_name>.s3-accelerate.amazonaws.com instead of s3.amazonaws.com, but unfortunately my team mates across the globe use IAM users that only have access to specific folders within the bucket, and I have not successfully connected when testing with those users. The way they are doing it now, they connect to the bucket (with the original S3 endpoint) and input a path in the path option to /<bucket_name>/path/to/access where path/to/access is the single folder they have access to in the bucket.

I use Cyberduck for my SFTP client and my team mates use WinSCP. I am not sure of the different options the tools have but I imagine the concepts are the same. Any ideas on how I can help them achieve faster speeds?

Answer

Hi,

Hope you are having a good day!

I understand you want to implement transfer acceleration for your users(Globally distributed) to access an S3 bucket located in US. Your approach is correct i.e. using .s3-accelerate.amazonaws.com instead of s3.amazonaws.com, however, the use-case does not require any SFTP client actually, you can achieve the same using AWS CLI only. The endpoint are HTTP/HTTPS endpoints and the client software will be interacting with them using HTTP protocol only (Not SFTP). Considering AWS CLI is available for most of the operating system and is consistent across multiple platforms, it would be a much better alternative.

—AWS CLI: https://aws.amazon.com/cli
—Installing AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

In AWS CLI, you can leverage “--endpoint-url” parameter to specify which endpoint the request should be sent. Also, you can edit your default profile to send all requests to transfer acceleration endpoints by default.You can read more about our supported method of using transfer acceleration in the below document,

—Using Transfer Acceleration: https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html

***AWS CLI reference example***
$ aws s3 cp file.txt s3://bucketname/keyname --region region --endpoint-url https://s3-accelerate.amazonaws.com
***
In addition to the above mentioned benefits, you can further optimize file performance of large uploads by altering parameters like maximum concurrent sessions,  etc. You can find details about these in the below document,

—How can I optimize performance when I upload large files to Amazon S3?: https://aws.amazon.com/premiumsupport/knowledge-center/s3-upload-large-files/

I will not be able to comment on the workings of WinSCP or Cyberduck in detail as I have very limited exposure of working on these 3rd party softwares. I hope you will understand. Having said that, I learned that these softwares directly access the S3 bucket and communicate using HTTPS endpoints only. I would like to know more about the reasoning of using this client since SFTP client are not required to access S3 buckets. If your intention is to list the bucket as a mountable drive or accessing the S3 bucket using “Transfer for SFTP” service (which accesses the S3 bucket as the back-end storage), kindly let me know. We can discuss a few other alternatives in details but first I would like to know more about the use-case.

What configuration do I need to use to utilize Transfer Acceleration when connecting to an S3 bucket via SFTP Client (Cyberduck / WinSCP) ?

相关内容