- Le plus récent
- Le plus de votes
- La plupart des commentaires
It depends on what exception caused your process to crash. Here's an article on setting up retries. I have the legacy method set up and I don't crash very often at all, but my internet does go out now and then. Let me know if this helps. Here's the article. I'm sure it applies to the aws cli, too.
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html
Also, since you're using multipart uploads, you might want to create a lifecycle run to save you some money with storage of failed multipart uploads. here's an article on that. https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/
Thank you. I didn't know I could create a lifecycle run for that. However, I've been running
aws s3api abort-multipart-upload
manually for each of the zombie multipart uploads that can be found inaws s3api list-multipart-uploads
:-) Thank you very much!
One more thing I should add. If you've been using multipart uploads in the past and you implement that lifecycle rule mentioned in my post, your costs per day may drop dramatically. Mine went from $2.50 per day to $0.31 per day. Open a case with AWS and tell them you were overcharged for incomplete multipart uploads that failed. They will ask you how far back you were using that, so be prepared to tell them that. You may get a refund going back that far. Good luck. I got a refund for $340 that way.
Hi There
You can also try a managed service like DataSync for migrating large datasets. DataSync takes care of the tuning and retries for you. Using the S3 CLI for large transfers can require a fair bit of system tuning. You’ll have to do some testing with the number of Amazon S3 concurrent threads, part size, possibly TCP window size, and likely parallel invocations of the S3 AWS CLI to match the throughput of AWS DataSync Take a look at this blog post for some other transfer methods you can try
https://aws.amazon.com/blogs/storage/migrating-and-managing-large-datasets-on-amazon-s3/
Contenus pertinents
- demandé il y a 15 jours
- demandé il y a 3 mois
- demandé il y a 5 mois
- AWS OFFICIELA mis à jour il y a 7 mois
- AWS OFFICIELA mis à jour il y a 2 ans
- AWS OFFICIELA mis à jour il y a 4 mois
- AWS OFFICIELA mis à jour il y a 8 mois
Thank you very much! I added now:
Hopefully this will resolve that :)