ECS 1.4 Pulling secrets and image

0

I using cloudformation and have followed the suggestion to adding endpoints to pull secrets and ecr images. However when I deploy my server/task it trys to provision but I get this:

STOPPED (ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 1 time(s): secrets manager: failed to retrieve secret from arn:aws:secretsmanag...)

Any insight on how to fix this?

Here is what my CF Template looks like:

SecretsManagerEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: ""
Principal: "
"
Resource: "*"
ServiceName: !Join [ "", [ "com.amazonaws.", { "Ref": "AWS::Region" }, ".secretsmanager" ] ]
VpcId: !Ref 'VPC'
VpcEndpointType: 'Interface'

ECRAPIEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: ""
Principal: "
"
Resource: "*"
ServiceName: !Join [ "", [ "com.amazonaws.", { "Ref": "AWS::Region" }, ".ecr.api" ] ]
VpcId: !Ref 'VPC'
VpcEndpointType: 'Interface'

Cloudformation also indicates these endpoints were created successfully.

BVM
asked 3 years ago2292 views
5 Answers
0

Hi BVM,

I’m work on the team that owns the code generating the error message you are seeing.

This error occurs when the Secrets Manager ARN fails format validation. I would recommend checking the ARN format. If that doesn’t resolve the problem please provide a list of task ARNs that are experiencing this issue and the AWS region for those task ARNs. That will allow us to look for the logs for the tasks.

We’ve taken an action item to improve the error messaging for this case.

Thanks,
Alex

answered 3 years ago
0

I am experiencing this same error. I was able to pull secrets from secrets manager on fargate platform version 1.3.0, but as soon as a redeploy with 1.4.0, I get this error. Any fix or troubleshooting steps suggestions? The arn for the secret in secrets manager is good as it worked previously with 1.3.0

mnjaws
answered 3 years ago
0

Is your task by any chance running in a private VPC? As described in the documentation and this blog[1], with PV 1.4.0, all task-related network traffic goes via the task ENI. In PV 1.3.0, Secrets Manager was accessed via the Fargate ENI.

[1] https://aws.amazon.com/blogs/containers/aws-fargate-launches-platform-version-1-4/

profile pictureAWS
Mats
answered 3 years ago
0

i am experience this same error as i am using fargate version 1.40. but i have other clusters using the same version of fargate and they are working properly. i have assigned more ECR READ permission to ecstaskexecution role but it didn't work.
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 1 time(s): InvalidParameterException: Invalid parameter at 'registryIds' fail..
any mistake i am making. i am making clusters and tasks using console UI.
any lead on solution ?

answered 3 years ago
0

Ah, thanks Alex! This helped me :)

I was getting this error as I tried to transition some of my team's code to retrieving individual JSON values from RDS database credentials secrets in Secrets Manager in an ECS task container definition (previously we'd been retrieving the entire secret JSON and parsing the values out using "jq"). Just like you'd suggested, my problem was that I hadn't formatted the ARN properly.

Note: the below code is part of a Terraform template file, but hope the problem/solution is still useful to others who may not be using Terraform ( https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file )

Our old code looked something like this:

...        
"secrets": \[  
  {  
    "name": "DB_CREDENTIALS",  
    "valueFrom": "${DB_CREDENTIALS_ARN}"  
  },  
  ...  

Then DB_CREDENTIALS would be an environment variable we could parse in a shell script. E.g.:

...
echo "Parsing credentials"
DB_HOST=$(echo "${DB_CREDENTIALS}" | jq -r .host)
DB_PORT=$(echo "${DB_CREDENTIALS}" | jq -r .port)
...

My first attempt at retrieving the host and port directly from secret JSON via the ARN looked like this:

...  
"secrets": \[  
  {  
    "name": "DB_HOST",  
    "valueFrom": "${DB_CREDENTIALS_ARN}:host"  
  },  
  {  
    "name": "DB_PORT",  
    "valueFrom": "${DB_CREDENTIALS_ARN}:port"  
  },  
  ...  

I was referencing this guide: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-secrets.html

Alex's reply helped me look back closer at the ARN and realize that even tho I'm not passing a version stage or version id to the secret ARN, I still need colons for them! Thus, I finally got things working with this:

...  
"secrets": \[  
  {  
    "name": "DB_HOST",  
    "valueFrom": "${DB_CREDENTIALS_ARN}:host::"  
  },  
  {  
    "name": "DB_PORT",  
    "valueFrom": "${DB_CREDENTIALS_ARN}:port::"  
  },  
  ...  

Edited by: pearcemerritt on Jul 25, 2021 11:29 PM

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions