Skip to content

AWS ECS will not start task from container on one cluster but will on others

0

I'm getting the following error:

CannotPullContainerError: ref pull has been retried 1 time(s): failed to unpack image on snapshotter overlayfs: mismatched image rootfs and manifest layers

Deployments are with CDK and I've deployed to several other ECS clusters without issue. I've torn everything down and recreated from scratch. I can run the generated containers with docker locally without issues. Cluster is using fargate. Images come from ECR. The infra is all infra as code with the CDK.

Any one have any ideas?

  • I'm afraid this is related with the way that the image is being generated/pushed. Could you please provide the commands that you are using to build and push the image to ECR? Thanks

  • @Henrique that would make sense if the image having issues on ECS didn't work on another ECS deployed with the same infra as code. When I used ECR + image on the second ECS it worked without issues.

  • I'm also able to pull the image from the ECR and run locally without any issues.

  • Got it! You are using CDK for creating the infrastructure. So, I'm assuming they are exactly the same. I'm also assuming that the ECR image is the same and you are testing on both.

    That said, it doesn't make much sense for me and I don't have any clue about what is going on. It should be something different between the both environment that would cause the issue. It is hard to say something without knowing your env. If you can Support contract, I'd encourage you to open a case to have this investigated. The Support team should be able to spot what is going on here looking at the envs.

asked 2 years ago1K views

2 Answers
0

I am facing a similar issue. I am deploying my infra using CDK and building the image from DockerFile locally on my machine using DockerImageAsset CDK construct. I manage to deploy the stack once.After updating the container, and redeploying the stack with CDK deploy, I can see that my ECS service update fails with this error. Nothing else has changed and the container is being built on the same machine in the same way.

AWS
EXPERT

answered 2 years ago

-1

The error you're encountering, "CannotPullContainerError: ref pull has been retried 1 time(s): failed to unpack image on snapshotter overlayfs: mismatched image rootfs and manifest layers," is a type of CannotPullContainer error in Amazon ECS. This specific error suggests there might be an issue with the image itself or how it's being pulled.

Given that you can run the containers locally and have successfully deployed to other ECS clusters, the problem might be related to the specific cluster or its configuration. Here are a few things you can check:

  1. Ensure that the task execution role associated with your Fargate tasks has the necessary permissions to pull images from your container registry (e.g., ECR).

  2. Verify that the cluster has proper network connectivity to pull the image. If you're using private subnets, make sure you have a NAT gateway configured correctly.

  3. Check if there are any Docker Hub rate limiting issues. If you're pulling images from Docker Hub, you might be hitting rate limits. You can authenticate Docker pulls for your container instances to increase the rate limit.

  4. Examine the VPC configuration of the problematic cluster. Ensure that it has the necessary routes to access the container registry endpoint.

  5. If you're using Amazon ECR, make sure your Docker client is authenticated with ECR for this specific cluster.

  6. Try pulling the image manually on an EC2 instance within the same VPC as your ECS cluster to isolate whether it's a network or permissions issue.

  7. Verify that the image tag in your task definition exactly matches the one in your container registry.

If none of these solutions work, you may need to investigate further by looking at the CloudWatch logs for your ECS tasks and the ECS agent logs on your container instances (if using EC2 launch type) for more detailed error messages.
Sources
CannotPullContainer task errors in Amazon ECS - Amazon Elastic Container Service

answered 2 years ago

  • I've updated the task on one of our other ECS's (that are deployed with the same CDK infra as code) to use the failing ECS's ECR and container. It works (I of course had update the role for the ECR so the task could pull). So none of these possible ideas seem valid.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.