ECR Behind an Nginx Proxy

Question

Hi Folks,

Due to network security requirements, we have a need to proxy requests to ECR through the following infrastructure

Client -> NLB -> EC2 (Nginx Proxy) -> VPCE -> ECR

We have the API call working to `aws ecr get-login-password` and we get the token, but the pipe to the docker login command returns:

`**.***.**.** - - [28/Nov/2023:02:38:22 +0000] "CONNECT *************.dkr.ecr.us-west-2.amazonaws.com:443 HTTP/1.1" 400 182 "-" "-"`

and from the client:

`Error response from daemon: Get "https://*************.dkr.ecr.us-west-2.amazonaws.com/v2/": Bad Request`

The nginx conf looks (mostly) like:

```
http {
    proxy_set_header    Host                $host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Proto   $scheme;

server {
        listen              443 ssl;
        server_name         "docker.dev.example.com";
        location / {
            proxy_pass           "https://*************.dkr.ecr.us-west-2.amazonaws.com";
        }
    }
}
```

I'm not quite sure where I can go next to see what may be missing. Does ECR have any logs to show what isn't there? This works fine outside the proxy from machines that are able to connect directly, so we have narrowed down the issue to the proxy config.

Thanks for any help you can provide.

Accepted Answer

To close the loop on this. My issue was that I was pointing the docker client at the proxy, as well as the proxying endpoint. This was unnecessary. But there were a few additional things that were needed:

* To log in and to access the repo, it is necessary to use the proxied domain, not the ECR domain, in the `docker login` call
* Once the login works, in order to pull, a proxy_redirect is necessary to write the URLs pointing to S3 for the layers. Ex:

`proxy_redirect      "https://prod-us-west-2-starport-layer-bucket.s3.us-west-2.amazonaws.com" "https://dkr-layer.example.com";`
* Ensure the Host header is rewritten to the ECR domain in the nginx proxy config

Hope this helps others. Cheers!

Answer

I understand that you need to proxy requests to ECR. Reviewing the nginx configuration, I see that you are delegating the requests to "xxxxxxx.dkr.ecr.us-west-2.amazonaws.com".

Please note that ECR has two different VPC Endpoints -- 
com.amazonaws.us-east-1.ecr.api
com.amazonaws.us-east-1.ecr.dkr

Ensure that you are passing requests to both of the ECR vpc endpoints. Coming to the error you are facing - "Bad Request" to ECR. Bad request error is usually observed when you pass the VPC Endpoint DNS name to the docker login command as shown in the example below:
-----------
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin vpce-xxxxxxxxxxxxxxx.dkr.ecr.us-west-2.vpce.amazonaws.com
------------

When setting up vpc endpoints for ECR, if you had  enabled "enable a private DNS hostname", then a private Route53 record is created in private host zones, which maps ".dkr.ecr..amazonaws.com" to private IP(s) of ECR VPC Endpoint. Therefore you will be able to use the below command

-----------
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin .dkr.ecr.us-west-2.vpce.amazonaws.com
------------

Unfortunately, we would not have ECR Logs that you can refer.

Considering, we need further information and Network specific details to be able to understand the exact reason for the error. Please feel free to open a support case with AWS Premium support to get further assistance.

ECR Behind an Nginx Proxy

Ensure that you are passing requests to both of the ECR vpc endpoints. Coming to the error you are facing - "Bad Request" to ECR. Bad request error is usually observed when you pass the VPC Endpoint DNS name to the docker login command as shown in the example below:

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin vpce-xxxxxxxxxxxxxxx.dkr.ecr.us-west-2.vpce.amazonaws.com

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.us-west-2.vpce.amazonaws.com

Relevant content