Backups on S3 storage not completing / taking too long

Question

We have several individual web servers backing up to S3 storage (individual buckets)
One of the smaller servers has 100 websites and backs up fully and quickly without any issues.
The other larger servers with roughly 300 plus websites each, backing up to separate buckets - these backups seem to take very long (many days) and still show incomplete - showing only half the amount of sites backed up.
What can be the issue - is it a timeout issue on our server or AWS setting or AWS plan to small or other ?

Answer

Did you check your upload speed on each server?  If the one(s) taking a long time have slower upload speeds or another process is using the bandwidth, then that could be your problem.

Answer

I would start with checking the transfer config, here are the docs for the CLI and boto3 (python). There will be similar docs for other SDKs as well.

https://docs.aws.amazon.com/cli/latest/topic/s3-config.html
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3.html

But also make sure you are using multipart uploads on bigger files.

Hope it helps!

Backups on S3 storage not completing / taking too long

Relevant content