CannotPullContainerError failed to copy: httpReadSeeker in Batch using Fargate

Question

My Batch job fails with the error CannotPullContainerError (failed to copy: httpReadSeeker). I'm using a private VCP and defined all the required Endpoints:

- com.amazonaws.eu-central-1.s3
- com.amazonaws.eu-central-1.ecr.api
- com.amazonaws.eu-central-1.ecr.dkr
- com.amazonaws.eu-central-1.logs
- com.amazonaws.eu-central-1.ecs
- com.amazonaws.eu-central-1.ecs-agent
- com.amazonaws.eu-central-1.ecs-telemetry

For the S3 endpoint, the policy is the default one:

```
{
	"Statement": [
		{
			"Action": "*",
			"Effect": "Allow",
			"Principal": "*",
			"Resource": "*"
		}
	]
}
```

The image is stored in a ECR private repository and exists (the process runs just fine when using Batch with EC2 instead of Fargate).

The Computing Env uses the Service role: AWSServiceRoleForBatch

The Batch job uses a custom executionRoleArn that has the standard policy AmazonECSTaskExecutionRolePolicy on it.

I have been looking in re:Post for similar errors and no solution works for me (when using a private ensure the use of endpoints, be sure that S3 endpoint has access to starport...).

Thanks,

David

Answer

Hello David,

The CannotPullContainerError you're experiencing seems to be associated with the inability of your Fargate task to pull the Docker image from your ECR repository.

Given that your VPC is private, the task needs a route to the internet to pull images from ECR. This route can be provided either through a NAT gateway or a VPC endpoint. It appears that you have set up the necessary endpoints for ECR and other services; however, pulling images from ECR also requires access to the S3 service, as ECR stores image layers in an S3 bucket, namely the "starport" bucket.

Please ensure that you have an S3 gateway endpoint configured in your VPC and that it has a policy allowing access to the "starport" bucket. The policy should look like this:

```
{
  "Statement": [
    {
      "Sid": "Access-to-specific-bucket-only",
      "Principal": "*",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": ["arn:aws:s3:::prod-region-starport-layer-bucket/*"]
    }
  ]
}
```
Please refer to the following document on how to set up an S3 gateway endpoint: https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html#ecr-setting-up-s3-gateway

Also, bear in mind that Fargate tasks running in private subnets will require a NAT gateway or a private subnet associated with a route table that has a default route to a NAT gateway or NAT instance to pull images. You might find this re:Post useful for more information on this topic: https://repost.aws/questions/QULIQs1kYOQKO1RpDaBkq-Wg/cannotpullcontainererror-in-the-private-network

Answer

Hi Ismail

The end point for S3 (com.amazonaws.eu-central-1.s3) is already in the VPC using a Full Access policy.  In any case I have expanded the Policy as:

```
{
	"Statement": [
		{
			"Action": "*",
			"Effect": "Allow",
			"Principal": "*",
			"Resource": "*"
		},
		{
			"Sid": "Access-to-specific-bucket-only",
			"Principal": "*",
			"Action": [
				"s3:GetObject"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::prod-eu-central-1-starport-layer-bucket/*"
			]
		}
	]
}
```

but the error is the same.

Thanks,

David

CannotPullContainerError failed to copy: httpReadSeeker in Batch using Fargate

Relevant content