AWS ECR undocumented error

0

I used to deploy my AWS ECS code with CDK with no problem. All of a sudden yesterday all tasks couldn't start. It shows Stopped status with following message on hover CannotPullContainerError: unsupported media type application/vnd.in-toto+json.

Is it fixable somehow? Now could not deploy anything.

UPDATE:

Before this deployment cdk deploy --all randomly fails with following error

failed commit on ref "manifest-sha256:851e5bb7dcd97948bd32720606df510feebee3ee9404e4db5e1ed00529803920": unexpected status from PUT request to https://408064982279.dkr.ecr.eu-west-1.amazonaws.com/v2/cdk-hnb659fds-container-assets-408064982279-eu-west-1/manifests/2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024: 400 Bad Request
EStreamingProcessor:  fail: docker push 408064982279.dkr.ecr.eu-west-1.amazonaws.com/cdk-hnb659fds-container-assets-408064982279-eu-west-1:2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024 exited with error code 1: failed commit on ref "manifest-sha256:851e5bb7dcd97948bd32720606df510feebee3ee9404e4db5e1ed00529803920": unexpected status from PUT request to https://408064982279.dkr.ecr.eu-west-1.amazonaws.com/v2/cdk-hnb659fds-container-assets-408064982279-eu-west-1/manifests/2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024: 400 Bad Request

 ❌ Deployment failed: Error: Failed to publish asset 2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024:408064982279-eu-west-1
    at Deployments.publishSingleAsset (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:11645)
    at async Object.publishAsset (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:197644)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:181545
[18:47:26] Reading cached notices from /Users/smotrov/.cdk/cache/notices.json

Failed to publish asset 2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024:408064982279-eu-west-1
[18:47:26] Error: Failed to publish asset 2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024:408064982279-eu-west-1
    at Deployments.publishSingleAsset (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:11645)
    at async Object.publishAsset (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:197644)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:181545

Right after this cdk deploy --all has no errors except not starting tasks.

1 Answer
1
Accepted Answer

Hello Smotrov,

The error suggests that there's an issue with pulling the container image from Amazon Elastic Container Registry (ECR). This error is likely related to a change in the media type being used by the container image manifest.

To resolve:

  1. Update Docker CLI and Engine: Make sure Docker is up to date.

  2. Specify Image Manifest Type: Use the following command to push the image with a compatible format:

    docker buildx build --platform linux/amd64,linux/arm64 --output "type=image,push=true,oci-mediatypes=false,name=your-repo:your-tag" .
  3. Update AWS CDK: Ensure your AWS CDK version is up to date.

  4. Rebuild and Push Image:

    docker build -t your-repo:your-tag .
    docker push your-repo:your-tag
  5. Review ECS Task Definition: Verify the image URI and credentials in your ECS task definition.

If the problem persists, please provide more information.

profile picture
EXPERT
answered a year ago
EXPERT
reviewed a year ago
profile picture
EXPERT
reviewed a year ago
  • Thank you for your prompt answer. But I do not doing docker build/push manually. It is done through CDK and CDK is a part of my CI/CD

  • Thanks for the information.

    You can try:

    1. Update AWS CDK:

      npm install -g aws-cdk@latest
    2. Configure Docker Image Format in CDK:

      const asset = new DockerImageAsset(this, 'MyBuildImage', {
        directory: path.join(__dirname, '../docker-image-directory'),
        platform: 'linux/amd64',
      });
      
      new ecs.ContainerImage.fromDockerImageAsset(asset);
    3. Check ECR Repository Configuration: Ensure your ECR repository supports the image types you are using.

    4. Review and Update CI/CD Pipeline: Ensure the pipeline uses the updated CDK configurations.

    5. Rebuild and Redeploy: Trigger a new build and deployment in your CI/CD pipeline.

    6. Check IAM Permissions: Ensure the IAM role used by the CI/CD pipeline has the necessary permissions for ECR.

    If the issue persists, please provide more details about your CDK and CI/CD setup.

  • Fixed with docker desktop reset / reinstall. Thank you so much!

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions