ECR login / docker login in private VPC not working

0

I really don't have much hope of getting answer on here but let's give it a go.

Docker login on my provate VPC instance has stopped working. I either get a timeout (Client.Timeout exceeded while awaiting headers) or context deadlin exceeded (Error response from daemon: Get "https://.... context deadline exceeded.

This working from my local machine. IAM permissions for the instance's role have not changed. NAT gateways is running and routing has not changed since it last worked

Dig gives some info

Local machine (account no not valid):

$ dig 123456789012.dkr.ecr.eu-west-1.amazonaws.com

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> 123456789012.dkr.ecr.eu-west-1.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53374
;; flags: qr rd ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;123456789012.dkr.ecr.eu-west-1.amazonaws.com. IN A

;; ANSWER SECTION:
123456789012.dkr.ecr.eu-west-1.amazonaws.com. 0 IN CNAME nlb3-3090dfc6de4af029.elb.eu-west-1.amazonaws.com.
nlb3-3090dfc6de4af029.elb.eu-west-1.amazonaws.com. 0 IN A 63.32.243.33

;; Query time: 30 msec
;; SERVER: 172.29.64.1#53(172.29.64.1) (UDP)
;; WHEN: Mon Apr 03 17:32:56 BST 2023
;; MSG SIZE  rcvd: 234

(Private) EC2 instance:

$ dig  123456789012.dkr.ecr.eu-west-1.amazonaws.com

; <<>> DiG 9.16.1-Ubuntu <<>> 123456789012.dkr.ecr.eu-west-1.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27919
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;206779301971.dkr.ecr.eu-west-1.amazonaws.com. IN A

;; ANSWER SECTION:
206779301971.dkr.ecr.eu-west-1.amazonaws.com. 12 IN A 10.1.1.76
206779301971.dkr.ecr.eu-west-1.amazonaws.com. 12 IN A 10.1.1.144

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Apr 03 16:18:53 UTC 2023
;; MSG SIZE  rcvd: 105

The most relevant, recent, other question on this forum here doesn't contain the answer (checked IAM permissions).

  • Update

    If I set the IP address in /etc/hosts to the public IP that I get when doing a dig from my local (laptop), login succeeds.

    i.e. in /etc/hosts, add: 63.32.243.33 111111111.dkr.ecr.eu-west-1.amazonaws.com

asked a year ago1315 views
1 Answer
0

Looks like there are a couple VPC interface endpoints setup in the VPC for ECR. Assuming that because the local dig returns local IPs. If thats true then check the security groups of the endpoints and instance, nacls of subnets, and endpoint policies. More info here: Amazon ECR interface VPC endpoints (AWS PrivateLink).

Returning the local IP is how those packets get routed to the VPC Interface endpoint and then to the ECR service, it is working as designed. Take a look at the VPC Reachability Analyzer to help find the blockage.

Also, consult AWS Config to see the timeline of any changes to the VPC configuration.

profile pictureAWS
EXPERT
kentrad
answered a year ago
  • Thanks for the response! Indeed, you are correct: There is a endpoint setup for erc.dkr in this VPC. However, the instance in question is not in the subnet that the endpoint is configured for. I would not expect it to get a (private) DNS record for these services. Also, this setup has been working for several months without this issue arising. Unless something changed and in AWS's backend and I need more config, like manually setting up a DHCP option set?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions