ECS CannotPullContainerError when using ECR Pull-Through-Cache rule


I configured a Docker-Hub pull-trough-cache rule in my ECR private repository.

When I use ECS to start a task, I get:

CannotPullContainerError: pull image manifest has been retried 1 time(s): 
failed to resolve ref <accountIdEDITED>.dkr.ecr.<regionEDITED> not found

But it works (without any change) after I perform a first pull via Docker CLI

% aws ecr get-login-password --profile <profileEDITED>  --region <regionEDITED> | docker login --username AWS --password-stdin <accountIdEDITED>.dkr.ecr.<regionEDITED>
Login Succeeded


 % docker pull <accountIdEDITED>.dkr.ecr.<regionEDITED>
fpm: Pulling from docker-hub/library/php
25d3892798f8: Pull complete 
52e7deb050f3: Pull complete 
1bba79b57e69: Pull complete 
0a396211340d: Pull complete 
bfdd050711cc: Pull complete 
369d0f9ad4eb: Pull complete 
7539dea78ec8: Pull complete 
2e4da698f1b3: Pull complete 
72ef6e71b468: Pull complete 
de30f3db8f79: Pull complete 

After I do this, I can see the repo in ECR UI.

My question: is there any way to get ECS to trigger repo creation via the Pull-Through-Cache rule without pulling manually once?

2 Answers
Accepted Answer

This is still very much a problem and not related to permissions, it is related to how ECS interfaces with ECR. It seems that under the hood it looks for image manifests of already available images in the repository, rather than doing your usual "docker pull" command. The "docker pull" command is what actually triggers ECR to pull the image, not fetching the manifest. It seems that ECS was simply not made to directly run any image via a pull-through cache.

It would be awesome if this could be fixed

answered 2 months ago
profile picture
reviewed a month ago

Ensure that your ECR repository has the correct permissions. The ECS task's IAM role or the EC2 instance profile associated with your ECS cluster should have permission to pull images from the ECR repository. Verify Docker-HubCache Configuration:Double-check your Docker-Hub pull-through cache configuration in ECR. Make sure it is correctly set up and associated with your ECR repository. If none of the above steps resolve it, consider reaching out to AWS support for more detailed assistance. They can help analyze specific logs and provide guidance based on the exact configuration and error messages you're encountering

profile picture
answered 3 months ago
  • The cluster has permission to pull from ECR. Indeed, after first manual pull, ECS manages to get the image from the ECR repo.

    My current set up uses AmazonECSTaskExecutionRolePolicy which provides BatchGetImage permission. Is there additional permission required to trigger ECR to actually pull the image from DockerHub?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions