ECR login / docker login in private VPC not working

Question

I really don't have much hope of getting answer on here but let's give it a go.

Docker login on my provate VPC instance has stopped working. I either get a timeout (Client.Timeout exceeded while awaiting headers) or context deadlin exceeded (Error response from daemon: Get "https://.... context deadline exceeded.

This working from my local machine. IAM permissions for the instance's role have not changed. NAT gateways is running and routing has not changed since it last worked

Dig gives some info

Local machine (account no not valid):

$ dig 123456789012.dkr.ecr.eu-west-1.amazonaws.com

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> 123456789012.dkr.ecr.eu-west-1.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53374
;; flags: qr rd ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;123456789012.dkr.ecr.eu-west-1.amazonaws.com. IN A

;; ANSWER SECTION:
123456789012.dkr.ecr.eu-west-1.amazonaws.com. 0 IN CNAME nlb3-3090dfc6de4af029.elb.eu-west-1.amazonaws.com.
nlb3-3090dfc6de4af029.elb.eu-west-1.amazonaws.com. 0 IN A 63.32.243.33

;; Query time: 30 msec
;; SERVER: 172.29.64.1#53(172.29.64.1) (UDP)
;; WHEN: Mon Apr 03 17:32:56 BST 2023
;; MSG SIZE  rcvd: 234

(Private) EC2 instance:

$ dig  123456789012.dkr.ecr.eu-west-1.amazonaws.com

; <<>> DiG 9.16.1-Ubuntu <<>> 123456789012.dkr.ecr.eu-west-1.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27919
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;206779301971.dkr.ecr.eu-west-1.amazonaws.com. IN A

;; ANSWER SECTION:
206779301971.dkr.ecr.eu-west-1.amazonaws.com. 12 IN A 10.1.1.76
206779301971.dkr.ecr.eu-west-1.amazonaws.com. 12 IN A 10.1.1.144

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Apr 03 16:18:53 UTC 2023
;; MSG SIZE  rcvd: 105

The most relevant, recent, other question on this forum here doesn't contain the answer (checked IAM permissions).

Answer

Looks like there are a couple VPC interface endpoints setup in the VPC for ECR. Assuming that because the local dig returns local IPs. If thats true then check the security groups of the endpoints and instance, nacls of subnets, and endpoint policies. More info here: Amazon ECR interface VPC endpoints (AWS PrivateLink).

Returning the local IP is how those packets get routed to the VPC Interface endpoint and then to the ECR service, it is working as designed. Take a look at the VPC Reachability Analyzer to help find the blockage.

Also, consult AWS Config to see the timeline of any changes to the VPC configuration.

Returning the local IP is how those packets get routed to the VPC Interface endpoint and then to the ECR service, it is working as designed. Take a look at the [VPC Reachability Analyzer](https://docs.aws.amazon.com/vpc/latest/reachability/getting-started.html) to help find the blockage.

Also, consult [AWS Config](https://docs.aws.amazon.com/config/latest/developerguide/getting-started.html) to see the timeline of any changes to the VPC configuration.

ECR login / docker login in private VPC not working

Dig gives some info

Update

Relevant content