docker push to ECR extremely slow

Question

Here are some reports that have the same issue as I have:
https://www.reddit.com/r/aws/comments/vsttg9/docker_push_to_ecr_very_slow/

My issue:
Usually docker push to the ECR registry is fast - however, sometimes that push is extremely slow (0.5 MB/sec or less). When this happened in the past a reboot of my system (ubuntu 22.04.3 lts) would have fixed that but not anymore (I upgraded docker recently and don't remember the prior version I had)
my docker version is Docker version 24.0.6, build ed223bc

I am pretty sure it is not related to linux, nor the docker version (as it happened with older versions as well; also see reports from others) and also happens from EC2 machines. Has anyone a solution for this?

Answer

There was plenty of free RAM, almost no CPU usage, plenty of bandwidth (12.5 GBit/sec connection). The issue is not instance resources.

Answer

As you cited correctly it does not seems related to linux and docker version.

There can be multiple factors while pushing the image, like instance resources, bandwidth etc. As reboot was solving issue, i suspect the resource crunch may be issue earlier.

However to check and narrow down, can you try checking with higher instance type (which comes with more bandwidth -> https://aws.amazon.com/ec2/instance-types/ bandwidth is mentioned for each instance type).

when push is slow, can you check instance metrics and instance state using TOP, free -m commands and see if there is resource crunch.

Answer

Hi,

I have faced this issue a couple of times, and the last time I did, I found that I was executing three VPN clients in background, none of them were actually connected to any VPN though.

Once I closed them, I restared Docker daemon, and the speed came back. Push was normal after that.

See if that's your case too.

Cheers.

--
Álvaro

docker push to ECR extremely slow

Relevant content