Skip to content

ECS task execution role is not using up-to-date policy

1

We have an ECS task that injects secrets from Secrets Manager, defined in a CloudFormation template created via the CDK. I have updated our CDK project to generate a new secret and inject it into the ECS task. The CloudFormation template has been updated to add the new secret's ARN to the task's execution role policy, allowing the "secretsmanager:GetSecretValue" action. The new secret is being generated successfully. When the ECS service is redeployed and the new containers are started, they fail with a similar error to the following

AccessDenied: User: arn:aws:sts::123456789012:assumed-role/SC-123456789012-pp-abcdef-mytask-abc123def/xyz987fed654cba321 is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-east-1:123456789012:secret:my-secret because no identity-based policy allows the secretsmanager:GetSecretValue action

This would be an otherwise very helpful error message if not for the fact that the IAM role mentioned in the message does in fact contain a statement allowing "secretsmanager:GetSecretValue" to be performed on the secret in question. If I open the IAM policy simulator and select the role, action and resource mentioned in the error message, I receive an "authorized" response. So, as far as I can see, there should be no issue with the ECS task accessing the secret. I'm not sure if this is the right place for this question, as this seems more likely to be a bug than a misconfiguration, but does anyone have any insight into why the task is failing to start?

3 Answers
1
Accepted Answer

This issue was finally resolved with the help of AWS technical support. The issue was that the secret reference in the task definition did not contain the full ARN of the secret, including the hyphen and additional six characters appended automatically by AWS. Since the secret name itself ended with a hyphen and six characters, this was being interpreted as the full ARN, not a partial ARN, which explains why other secrets were not experiencing the same issue.

Below is the full explanation provided by AWS technical support:

The full ARN format for a Secrets Manager secret is usually in this format:

arn:aws:secretsmanager:region:aws_account_id:secret:secret-name-a1b2c3:version-stage:version-id

Where:

  • region is the AWS region.
  • aws_account_id is your AWS account ID.
  • secret-name-a1b2c3 is the name of the secret appended with a unique suffix (-a1b2c3) that AWS automatically generates.The suffix is a hyphen followed by six random characters (e.g., -a1b2c3).
  • version-stage and version-id are optional and used for versioning.

When you create a secret in Secrets Manager, AWS appends a unique suffix (e.g., -a1b2c3) to the secret name you provide. This suffix ensures that the secret name is unique within your account. This is particularly useful when secrets are deleted and recreated with the same name.

Please note that when referencing secrets, you can often use a partial ARN (without the suffix) or the full ARN (including the suffix). However, there's an important exception:

[1] If your secret name ends with a hyphen followed by six characters, Secrets Manager might interpret a partial ARN as a full ARN. In this case, you must use the full ARN including the suffix. In other words, If your secret name ends with a hyphen and six characters, using only part of the ARN can appear to Secrets Manager as though you are specifying a full ARN. For example: you might have a secret named MySecret-abcdef with the ARN

arn:aws:secretsmanager:us-east-2:111122223333:secret:MySecret-abcdef-nutBrk

If you call the following operation, which only uses part of the secret ARN, then Secrets Manager might not find the secret.

$ aws secretsmanager describe-secret --secret-id arn:aws:secretsmanager:us-east-2:111122223333:secret:MySecret-abcdef

Hence, The reason your existing task definitions work without the suffix while a new one requires it would be either due to:

a. The names of your existing secrets don't end with a hyphen and six characters.

b. You're using a different method to reference the secrets that doesn't require the full ARN.

To ensure consistency and avoid potential issues, it's recommended to always use the full ARN, including the suffix, when referencing secrets in your task definitions.

To find the full ARN of a secret, you can use the AWS CLI command:

aws secretsmanager describe-secret --secret-id secret-name This will return the full ARN, which you can then use in your task definitions.

By using the full ARN consistently, you can avoid potential issues and ensure that your task definitions work correctly for all secrets, regardless of their naming convention.

answered 2 years ago

EXPERT

reviewed 2 years ago

0

Also ensure the policy role has KMS Decrypt if using a customer managed key as it will give you the same error if the secret is encrypted with a customer managed key and it doesnt have the decrypt permission!

EXPERT

answered 2 years ago

  • Thanks Gary, the secret is using the AWS managed KMS key, so not relevant for this case unfortunately.

  • Are you injecting them into the container as environment variables or is it your code pulling secrets within the task?

  • As environment variables.

  • I updated the policy and waited overnight before trying to redeploy the containers and still no luck.

0

The issue you're experiencing with your ECS task failing to access the new secret despite having the correct IAM permissions is likely due to a delay in policy propagation or caching. Here are a few potential reasons and solutions to consider:

  1. IAM Policy Propagation Delay: Changes to IAM policies can take some time to propagate across all AWS regions and services. This delay can be up to several minutes or, in rare cases, hours. Even though the policy has been updated in the IAM console or through CloudFormation, it may not have fully propagated to the ECS service yet.

  2. ECS Task Execution Role Caching: The ECS service might be caching the task execution role's permissions. When you update the role's permissions, the cached version may still be in use for a period of time.

  3. Container Agent Caching: The ECS container agent on the EC2 instances (if you're not using Fargate) might be caching the task execution role credentials.

To resolve this issue, you can try the following:

  1. Wait and Retry: Give it some time (15-30 minutes) and try redeploying your ECS service again. This allows time for the policy changes to propagate fully.

  2. Force New Deployment: In the ECS console or using the AWS CLI, force a new deployment of your ECS service. This can help refresh the cached permissions.

  3. Restart ECS Instances: If you're using EC2 launch type, consider restarting your ECS container instances to force a refresh of the cached credentials.

  4. Verify Secret ARN: Double-check that the exact ARN of the secret in your IAM policy matches the ARN being used in your task definition. Even small discrepancies can cause access issues.

  5. Check for Conflicting Policies: Ensure there are no other policies attached to the role or the ECS task that might be conflicting with or overriding the permissions you've set.

  6. Use the Latest ECS Agent: If you're using the EC2 launch type, make sure your ECS container agent is up to date, as older versions might have issues with permission refreshing.

If the issue persists after trying these solutions, it may indeed be a bug, and you should consider contacting AWS support for further assistance.
Sources
Amazon ECS task execution IAM role - Amazon Elastic Container Service
Determine who has permissions to your AWS Secrets Manager secrets - AWS Secrets Manager

answered 2 years ago

EXPERT

reviewed 2 years ago

  • Policy propagation delay and role caching are good suggestions, but I've updated the policy and waited over 30 minutes before redeploying the ECS service and still no luck.

    The service is using Fargate so the EC2 suggestions are not relevant in this case.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.