Cost of Moving Files

Question

We are in Process of Moving around 60M files from s3 bucket to local filesystem and we wanted to understand the best way to accomplish this in terms of cost

We were thinking of the below options

1)Copy files from S3 to Local using aws cp command
2)Use Aws S3 sync command 
3)Move files from  S3 to Local using aws mv command

Additional we also wanted to completely delete the files with versions from the S3 bucket after certain days (Example 30 days)

Currently we were thinking move would be cost effective but needed some inputs on this

Answer

When moving data from S3 to local, you will be paying two parts: 1) COPY request which is about $0.005 per 1,000 request 2) data transfer which is about $0.05/GB when it's greater than 150TB/mo. You can find more details on the [pricing](https://aws.amazon.com/s3/pricing) page. It wouldn't make a difference from cost perspective between move vs copy since Delete requests are free. You would end up paying both COPY request and Data Transfer.

Answer

Hi,

Since you would like to download data from S3, it will incur an egress cost. The method of downloading this data wouldn't influence the amount of data downloaded. Pls refer this [blog](https://aws.amazon.com/blogs/apn/aws-data-transfer-charges-for-server-and-serverless-architectures/) to understand this concept.

The only thing which comes to my mind is you could consider compressing the files before downloading i.e. if you receive a good compression ratio. You can do this using an EC2 or Lambda, refer this post for an [approach](https://repost.aws/questions/QUClJ2dmvbQJKje4LI_Gj2-w/to-convert-all-files-in-an-aws-s3-bucket-to-zip-files). Worth testing it out with a few files, since you will spend on compute to compress these files too.

Hope this is helpful!

Thanks,
Rama

Cost of Moving Files

関連するコンテンツ