By using AWS re:Post, you agree to the Terms of Use
/CannotPullContainerError in public VPC/Subnet. What am I missing/doing wrong?/

CannotPullContainerError in public VPC/Subnet. What am I missing/doing wrong?

0

I have created a brand new AWS account (just to troubleshoot this issue) and the default VPC and subnets of every region in this account are left pristine and unmodified.

Here's the default VPC in us-east-1:

$ aws ec2 describe-vpcs
{
    "Vpcs": [
        {
            "CidrBlock": "172.31.0.0/16",
            "DhcpOptionsId": "dopt-095a7873b289557a1",
            "State": "available",
            "VpcId": "vpc-08ba51697a37c5ad9",
            "OwnerId": "...",
            "InstanceTenancy": "default",
            "CidrBlockAssociationSet": [
                {
                    "AssociationId": "vpc-cidr-assoc-0dba5df7b176877b7",
                    "CidrBlock": "172.31.0.0/16",
                    "CidrBlockState": {
                        "State": "associated"
                    }
                }
            ],
            "IsDefault": true
        }
    ]
}

Here's the route table for this VPC:

$ aws ec2 describe-route-tables --filters Name=vpc-id,Values=vpc-08ba51697a37c5ad9
{
    "RouteTables": [
        {
            "Associations": [
                {
                    "Main": true,
                    "RouteTableAssociationId": "rtbassoc-08e6f9833f341f6c4",
                    "RouteTableId": "rtb-000d61d5d0236d276",
                    "AssociationState": {
                        "State": "associated"
                    }
                }
            ],
            "PropagatingVgws": [],
            "RouteTableId": "rtb-000d61d5d0236d276",
            "Routes": [
                {
                    "DestinationCidrBlock": "172.31.0.0/16",
                    "GatewayId": "local",
                    "Origin": "CreateRouteTable",
                    "State": "active"
                },
                {
                    "DestinationCidrBlock": "0.0.0.0/0",
                    "GatewayId": "igw-0b7ed209f5cd38fa6",
                    "Origin": "CreateRoute",
                    "State": "active"
                }
            ],
            "Tags": [],
            "VpcId": "vpc-08ba51697a37c5ad9",
            "OwnerId": "..."
        }
    ]
}

As you can see, the second route permits egress to the internet:

{
    "DestinationCidrBlock": "0.0.0.0/0",
    "GatewayId": "igw-0b7ed209f5cd38fa6",
    "Origin": "CreateRoute",
    "State": "active"
}

So I assume if I deploy an ECS Fargate task in this VPC, it should be able to pull amazoncorretto:17-alpine3.15 from docker.io.

Despite that, whenever I deploy my CloudFormation stack, ECS fails to run the scheduled task as it cannot fetch the images from DockerHub and prints this error:

CannotPullContainerError: inspect image has been retried 5 time(s): failed to resolve ref "docker.io/library/amazoncorretto:17-alpine3.15": failed to do request: Head https://registry-1.docker.io/v2/library/amazoncorretto/manifests/17-alpine3.15: dial ...

Here's my CloudFormation template (I have intentionally given wide open permissions to all the roles involved to make sure this issue is not due to insufficient IAM permissions):

AWSTemplateFormatVersion: "2010-09-09"
Description: ECS Cron Task
Parameters:
  AppName:
    Type: String
    Default: CronTask

  AppImage:
    Type: String
    Default: amazoncorretto:17-alpine3.15

  AppLogGroup:
    Type: String
    Default: ECS

  AppLogPrefix:
    Type: String
    Default: CronTask

  ScheduledTaskSubnets:
    Type: List<AWS::EC2::Subnet::Id>
    Default: "subnet-0031a6eaf7e52173c, subnet-01950a0d2d1e04dc1, subnet-0a1aa70f0421e2025, subnet-036abb95995a86c73, subnet-0f8b5043babfb9a7e, subnet-07cb2210ce2d5bb8f"

Resources:
  Cluster:
    Type: AWS::ECS::Cluster

  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
      Policies:
        - PolicyName: AdminAccess
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action: "*"
                Effect: Allow
                Resource: "*"

  TaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
      Path: /
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
        - PolicyName: AdminAccess
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action: "*"
                Effect: Allow
                Resource: "*"

  TaskScheduleRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: events.amazonaws.com
      Path: /
      Policies:
        - PolicyName: AdminAccess
          PolicyDocument:
            Statement:
              - Action: "*"
                Effect: Allow
                Resource: "*"

  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 256
      Memory: 512
      NetworkMode: awsvpc
      TaskRoleArn: !Ref TaskRole
      ExecutionRoleArn: !Ref TaskExecutionRole
      Family: !Ref AppName
      RequiresCompatibilities:
        - FARGATE
      ContainerDefinitions:
        - Name: !Ref AppName
          Image: !Ref AppImage
          Command: ["java", "--version"]
          Essential: true
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-create-group: true
              awslogs-group: !Ref AppLogGroup
              awslogs-region: !Ref "AWS::Region"
              awslogs-stream-prefix: !Ref AppLogPrefix

  TaskSchedule:
    Type: AWS::Events::Rule
    DependsOn: 
      - TaskScheduleRole
      - DeadLetterQueue
    Properties:
      Description: Trigger the task once every minute
      ScheduleExpression: cron(0/1 * * * ? *)
      State: ENABLED
      Targets:
        - Arn: !GetAtt Cluster.Arn
          Id: ClusterTarget
          RoleArn: !GetAtt TaskScheduleRole.Arn
          DeadLetterConfig:
            Arn: !GetAtt DeadLetterQueue.Arn
          EcsParameters:
            LaunchType: FARGATE
            TaskCount: 1
            TaskDefinitionArn: !Ref TaskDefinition
            NetworkConfiguration:
              AwsVpcConfiguration:
                Subnets: !Ref ScheduledTaskSubnets

  DeadLetterQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: "CronTaskDeadLetterQueue"

  DeadLetterQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref DeadLetterQueue
      PolicyDocument:
        Statement:
          - Action: "*"
            Effect: Allow
            Resource: "*"

What am I missing here? Why despite running the task in a public subnet/VPC (below), AWS cannot pull the image from docker.io? Is something missing in my TaskSchedule resource?

TaskSchedule:
    Type: AWS::Events::Rule
    ...
    Properties:
        ...
        Targets:
            - ...
                EcsParameters:
                LaunchType: FARGATE
                TaskCount: 1
                TaskDefinitionArn: !Ref TaskDefinition
                NetworkConfiguration:
                    AwsVpcConfiguration:
                        Subnets: !Ref ScheduledTaskSubnets

Thanks in advance.

1 Answers
1
Accepted Answer

Your Task is unable to communicate with the Internet due to not being assigned a public IP address. You need to add AssignPublicIp: ENABLED in your AwsVpcConfiguration. See also https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_AwsVpcConfiguration.html.

answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions