Uploading small files via WinSCP is very slow

Question

I am new to AWS EC2 and have a problem that has been bothering me for a long time now, and I have tried a number of solutions, such as changing the credit specification to unlimited mode、increasing gp3's IOPS and Thoughput, but all to no avail!

I'm trying to upload a small file with 200M of data to EC2 via WinSCP and it takes about 30 minutes, but if I pack it into a single file it takes only 2.5 minutes.

However, on the previous provider's server (not AWS), uploading a small file with 200M of data takes about 5 minutes, and if it's packed into a single file, it takes about 3 minutes.

Why is there such a big difference when uploading small files to EC2? I understand that small files can be slow to transfer efficiently, but 30 minutes seems a bit unbelievable.

Is there a way to improve the transfer efficiency of small files without packing them into a single file?

Answer

Hi

**Compress:** Before transfer, compress the smaller files into a single archive (e.g., .zip, .tar.gz). This will:

* Reduce the total number of files being transferred, minimizing overhead.
* Improve the overall transfer speed.

Consider file transfer tools other than WinSCP that might be better optimized for numerous small transfers:

**rsync**: Highly efficient in synchronizing directories and minimizes unnecessary transfers. Ideal if you repeatedly update your files.

**Linux:**

`rsync -avz source_directory user@ec2_instance_ip:/destination_directory`

If you are using Keypair

`rsync -avz -e "ssh -i /path/to/your/private_key" source_directory user@EC2_IP_address:/destination_directory`

**Windows:**

Install Cygwin -- Install rsync via Cygwin's setup --Open Cygwin Terminal and use rsync as in the Linux example.

**S3 Upload:**  - This is the Best Part I can say, Information [https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html]( Documentation Link)
Use the AWS CLI or SDKs to upload to an S3 bucket and then copy the file into your EC2 instance. This leverages S3's parallelization for large data sets.

Uploading small files via WinSCP is very slow

Relevant content