- Newest
- Most votes
- Most comments
This issue was finally resolved with the help of AWS technical support. The issue was that the secret reference in the task definition did not contain the full ARN of the secret, including the hyphen and additional six characters appended automatically by AWS. Since the secret name itself ended with a hyphen and six characters, this was being interpreted as the full ARN, not a partial ARN, which explains why other secrets were not experiencing the same issue.
Below is the full explanation provided by AWS technical support:
The full ARN format for a Secrets Manager secret is usually in this format:
arn:aws:secretsmanager:region:aws_account_id:secret:secret-name-a1b2c3:version-stage:version-id
Where:
- region is the AWS region.
- aws_account_id is your AWS account ID.
- secret-name-a1b2c3 is the name of the secret appended with a unique suffix (-a1b2c3) that AWS automatically generates.The suffix is a hyphen followed by six random characters (e.g., -a1b2c3).
- version-stage and version-id are optional and used for versioning.
When you create a secret in Secrets Manager, AWS appends a unique suffix (e.g., -a1b2c3) to the secret name you provide. This suffix ensures that the secret name is unique within your account. This is particularly useful when secrets are deleted and recreated with the same name.
Please note that when referencing secrets, you can often use a partial ARN (without the suffix) or the full ARN (including the suffix). However, there's an important exception:
[1] If your secret name ends with a hyphen followed by six characters, Secrets Manager might interpret a partial ARN as a full ARN. In this case, you must use the full ARN including the suffix. In other words, If your secret name ends with a hyphen and six characters, using only part of the ARN can appear to Secrets Manager as though you are specifying a full ARN. For example: you might have a secret named MySecret-abcdef with the ARN
arn:aws:secretsmanager:us-east-2:111122223333:secret:MySecret-abcdef-nutBrkIf you call the following operation, which only uses part of the secret ARN, then Secrets Manager might not find the secret.
$ aws secretsmanager describe-secret --secret-id arn:aws:secretsmanager:us-east-2:111122223333:secret:MySecret-abcdef
Hence, The reason your existing task definitions work without the suffix while a new one requires it would be either due to:
a. The names of your existing secrets don't end with a hyphen and six characters.
b. You're using a different method to reference the secrets that doesn't require the full ARN.
To ensure consistency and avoid potential issues, it's recommended to always use the full ARN, including the suffix, when referencing secrets in your task definitions.
To find the full ARN of a secret, you can use the AWS CLI command:
aws secretsmanager describe-secret --secret-id secret-name This will return the full ARN, which you can then use in your task definitions.
By using the full ARN consistently, you can avoid potential issues and ensure that your task definitions work correctly for all secrets, regardless of their naming convention.
Also ensure the policy role has KMS Decrypt if using a customer managed key as it will give you the same error if the secret is encrypted with a customer managed key and it doesnt have the decrypt permission!
The issue you're experiencing with your ECS task failing to access the new secret despite having the correct IAM permissions is likely due to a delay in policy propagation or caching. Here are a few potential reasons and solutions to consider:
-
IAM Policy Propagation Delay: Changes to IAM policies can take some time to propagate across all AWS regions and services. This delay can be up to several minutes or, in rare cases, hours. Even though the policy has been updated in the IAM console or through CloudFormation, it may not have fully propagated to the ECS service yet.
-
ECS Task Execution Role Caching: The ECS service might be caching the task execution role's permissions. When you update the role's permissions, the cached version may still be in use for a period of time.
-
Container Agent Caching: The ECS container agent on the EC2 instances (if you're not using Fargate) might be caching the task execution role credentials.
To resolve this issue, you can try the following:
-
Wait and Retry: Give it some time (15-30 minutes) and try redeploying your ECS service again. This allows time for the policy changes to propagate fully.
-
Force New Deployment: In the ECS console or using the AWS CLI, force a new deployment of your ECS service. This can help refresh the cached permissions.
-
Restart ECS Instances: If you're using EC2 launch type, consider restarting your ECS container instances to force a refresh of the cached credentials.
-
Verify Secret ARN: Double-check that the exact ARN of the secret in your IAM policy matches the ARN being used in your task definition. Even small discrepancies can cause access issues.
-
Check for Conflicting Policies: Ensure there are no other policies attached to the role or the ECS task that might be conflicting with or overriding the permissions you've set.
-
Use the Latest ECS Agent: If you're using the EC2 launch type, make sure your ECS container agent is up to date, as older versions might have issues with permission refreshing.
If the issue persists after trying these solutions, it may indeed be a bug, and you should consider contacting AWS support for further assistance.
Sources
Amazon ECS task execution IAM role - Amazon Elastic Container Service
Determine who has permissions to your AWS Secrets Manager secrets - AWS Secrets Manager
Policy propagation delay and role caching are good suggestions, but I've updated the policy and waited over 30 minutes before redeploying the ECS service and still no luck.
The service is using Fargate so the EC2 suggestions are not relevant in this case.
Relevant content
asked 2 years ago
asked 3 years ago

Thanks Gary, the secret is using the AWS managed KMS key, so not relevant for this case unfortunately.
Are you injecting them into the container as environment variables or is it your code pulling secrets within the task?
As environment variables.
I updated the policy and waited overnight before trying to redeploy the containers and still no luck.