ECR Cross Account Private Link

0

Hi All, Struggling for a couple of days already with the following: I've followed this guide: https://aws.amazon.com/blogs/networking-and-content-delivery/centralize-access-using-vpc-interface-endpoints/ I have setup AWS Organisations with all the separate Accounts like nonprd, prd, .... AND the Shared resources account.... CIDR for Shared: 10.40.0.0/16 CIDR for nonprd: 10.0.0.0/16 CIDR for prd: 10.1.0.0/16 In this shared resources account, I've created the 4 vpc endoints for ECR (Shared resources account holds our ecr docker repos for other accounts). logs,dkr,api and S3. I've setup VPC peering with my nonprd and prd account. I've created the route table entries so that all traffic is flowing from shared to the vpc-peering connections cidr and visa versa. The private dns option for the VPC Endpoints are disabled and manually created as private Route53 records. Exactly as the ecr domain. So I have 3 extra private records IN the SHARED resources account:

  • api.ecr.eu-west-1.amazonaws.com
  • dkr.ecr.eu-west-1.amazonaws.com
  • logs.eu-west-1.amazonaws.com

I've create the alias record pointing to the private hosted zones. I've done the Associations for Route53 from all the VPC's in nonprd and prd. I CAN resolve the dns records. BUT... And now the problem arises... When I try to run the containers in my nonprd account in any of the vpc's there, my tasks are given one of the following errors:

  • ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 1 time(s): AccessDeniedException: User: arn:aws:sts::${AWS::AccountId}:assumed-rol...
  • ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post https://api.ecr....

The policy on the VPC endpoints (complete snippet from my cfn-template):

EcrApiEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              AWS:
                - "arn:aws:iam::51*******0:root" # NonPrd
                - "arn:aws:iam::1******3:root" # Prd
                - !Sub arn:aws:iam::${AWS::AccountId}:root
            Action:
              - ecr:BatchGetImage
              - ecr:GetAuthorizationToken
              - ecr:GetDownloadUrlForLayer
              - ecr:BatchCheckLayerAvailability
              - ecr:PutImage
              - ecr:InitiateLayerUpload
              - ecr:UploadLayerPart
              - ecr:CompleteLayerUpload
            Resource:
              - !Sub "arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/*"
      VpcId: !FindInMap [Environments, !Ref Environment, VPC]
      VpcEndpointType: Interface
      PrivateDnsEnabled: false
      SecurityGroupIds:
        - !GetAtt VPCESecurityGroup.GroupId
      SubnetIds:
        - !Select [
            0,
            !FindInMap [Environments, !Ref Environment, PrivateSubnets],
          ]
        - !Select [
            1,
            !FindInMap [Environments, !Ref Environment, PrivateSubnets],
          ]
        - !Select [
            2,
            !FindInMap [Environments, !Ref Environment, PrivateSubnets],
          ]
      ServiceName:
        Fn::Join:
          - ""
          - - "com.amazonaws."
            - !Ref "AWS::Region"
            - ".ecr.api"

So the VPC endpoints run in the Private subnet of the SHARED resources account. The ecs fargate service/task also has the correct permissions (everything is working fine without the VPC endpoints). Can someone help... Please...

4 Answers
0

Can you share a screenshot or information for Route53 Hosted Zone and Record how currently configured for PrivateLink's domain setup? Additionally, I would like to check the security group rules connected to PrivateLink.

profile picture
EXPERT
answered 2 years ago
0

Hi, Thanks for helping. Yes. I have everything setup in cfn so...

This is one of the Hosted Private zones (the dkr one, created in the SHARED resources account, where of course the ecr repos live):

PrivateHostedZoneEcrDkr:
    Type: "AWS::Route53::HostedZone"
    Properties:
      HostedZoneConfig:
        Comment: "Private hosted zone for ecr  dkr vpc endpoint"
      Name: "dkr.ecr.eu-west-1.amazonaws.com"
      VPCs:
        - VPCId: !FindInMap [Environments, !Ref Environment, VPC]
          VPCRegion: !Sub ${AWS::Region}
      HostedZoneTags:
        - Key: ApplicationName
          Value: !Ref ApplicationName
        - Key: Availability
          Value: !Ref Availability
        - Key: ContactPerson
          Value: !Ref ContactPerson
        - Key: Environment
          Value: !Ref Environment
        - Key: Monitor
          Value: !Ref Monitor
        - Key: ProjectName
          Value: !Ref ProjectName

The security group rules for the private link(dkr) and private link setup:

EcrDkrEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              AWS:
                - "arn:aws:iam::5*********0:root" # NonPrd
                - "arn:aws:iam::1*********3:root" # Prd
                - !Sub arn:aws:iam::${AWS::AccountId}:root
            Action:
              - ecr:BatchGetImage
              - ecr:GetAuthorizationToken
              - ecr:GetDownloadUrlForLayer
              - ecr:BatchCheckLayerAvailability
              - ecr:PutImage
              - ecr:InitiateLayerUpload
              - ecr:UploadLayerPart
              - ecr:CompleteLayerUpload
            Resource:
              - !Sub "arn:aws:ecr:${AWS::Region}:289334807197:repository/*"
      VpcId: !FindInMap [Environments, !Ref Environment, VPC]
      VpcEndpointType: Interface
      PrivateDnsEnabled: false
      SecurityGroupIds:
        - !GetAtt VPCESecurityGroup.GroupId
      SubnetIds:
        - !Select [
            0,
            !FindInMap [Environments, !Ref Environment, PrivateSubnets],
          ]
        - !Select [
            1,
            !FindInMap [Environments, !Ref Environment, PrivateSubnets],
          ]
        - !Select [
            2,
            !FindInMap [Environments, !Ref Environment, PrivateSubnets],
          ]
      ServiceName:
        Fn::Join:
          - ""
          - - "com.amazonaws."
            - !Ref "AWS::Region"
            - ".ecr.dkr"

VPCSecurityGroup:

VPCESecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub "vpc-endpoint-sg-${Environment}"
      GroupDescription: "VPCE endpoint security group"
      VpcId: !FindInMap [Environments, !Ref Environment, VPC]
      SecurityGroupIngress:
        - CidrIp: !FindInMap [Environments, !Ref Environment, CIDR]
          FromPort: 443
          IpProtocol: tcp
          ToPort: 443
        - CidrIp: !FindInMap [Environments, "prd", CIDR]
          FromPort: 443
          IpProtocol: tcp
          ToPort: 443
        - CidrIp: !FindInMap [Environments, "staging", CIDR]
          FromPort: 443
          IpProtocol: tcp
          ToPort: 443
        - CidrIp: !FindInMap [Environments, "qa", CIDR]
          FromPort: 443
          IpProtocol: tcp
          ToPort: 443
answered 2 years ago
0

One more important thing probably. If I try to do a get-login on a ec2 I get this: is not authorized to perform: ecr:GetAuthorizationToken on resource: * because no VPC endpoint policy allows the ecr:GetAuthorizationToken action However the ecr:GetAuthorizationToken is in the policy of the endpoints:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Action": [
				"ecr:BatchGetImage",
				"ecr:GetAuthorizationToken",
				"ecr:GetDownloadUrlForLayer",
				"ecr:BatchCheckLayerAvailability",
				"ecr:PutImage",
				"ecr:InitiateLayerUpload",
				"ecr:UploadLayerPart",
				"ecr:CompleteLayerUpload"
			],
			"Resource": [
				"arn:aws:ecr:eu-west-1:289334807197:repository/*"
			],
			"Effect": "Allow",
			"Principal": {
				"AWS": [
					"arn:aws:iam::5***************0:root",
					"arn:aws:iam::1***************3:root",
					"arn:aws:iam::2***************7:root"
				]
			}
		}
	]
}
answered 2 years ago
0

I've tried to add the SG for the Fargate cluster to the VPC Endpoints... Still no luck... I'm out of ideas fot the moment... So... Anyone?

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions