Amazon ECS container permissions issue

Question

I have a container I'm trying to run on ECS.  I'm getting this error message:

```
2025-03-05T14:50:51.765Z  ***************************
2025-03-05T14:50:51.765Z  APPLICATION FAILED TO START
2025-03-05T14:50:51.765Z  ***************************
2025-03-05T14:50:51.765Z  Description:
2025-03-05T14:50:51.765Z  Config data resource '[SecretsManagerConfigDataResource@54ac9be7 context = 'myapp/staging/credentials', optional = false, enabled = true]' via location  'aws-secretsmanager:myapp/staging/credentials' does not exist
2025-03-05T14:50:51.765Z  Action:
2025-03-05T14:50:51.765Z  Check that the value 'aws-secretsmanager:myapp/staging/credentials' at class path resource [application-staging.properties] - 4:22 is correct, or prefix it with 'optional:'
```

So it can't find the SSM secret.  However, I know the secret exists because I can run my code from my local system and it finds it and starts up without any issue.  So I'm left to believe this is some sort of permission issue.

I've assigned the following Roles to my Task Definition:
Task role: **myapp_staging_task**

**Permissions:**
* From the default policy: **AmazonSSMManagedInstanceCore**
* From the default policy: **CloudWatchAgentServerPolicy**
* Then some custom permissions that don't have anything to do with SSM.

Task execution role: **ecsTaskExecutionRole**

**Permissions**:
* From the Default policy: **AmazonECSTaskExecutionRolePolicy**

The Ec2 instance the container is running on has a Role of: **ecsInstanceRole**

**Permissions**:
* From the default policy: **AmazonEC2ContainerServiceforEC2Role**

So I'm granting it access to SSM as far as I can tell. Unless the Task Role and Task Execution Role are backwards.  I've read the docs, and while I think I understand them.  They are tremendously confusing, and the names really really don't help.

So why can't my container see SSM?  Or how else can I trouble shoot this efficiently?

re:Post Agent · Accepted Answer

Based on the error message and the information provided, it appears that your Amazon ECS container is unable to access the secret stored in AWS Secrets Manager, not AWS Systems Manager (SSM) as you mentioned. The error specifically refers to "aws-secretsmanager:myapp/staging/credentials", which indicates it's trying to access a secret in Secrets Manager.

To resolve this issue, you need to ensure that the correct IAM role has the necessary permissions to access Secrets Manager. In this case, the Task Role (myapp_staging_task) should have the required permissions, not the Task Execution Role.

Here are the steps to troubleshoot and resolve the issue:

1. Add the necessary permissions to the Task Role (myapp_staging_task):
   - Add the "secretsmanager:GetSecretValue" permission to this role.
   - Ensure that the ARN of the secret is correctly specified in the IAM policy.

2. Verify that the IAM policy is attached to the correct role (myapp_staging_task).

3. Double-check that you're using the correct AWS Region when trying to access the secret. The Region should match the one where your secret is stored.

4. To test if the permissions are set correctly, you can try running the following AWS CLI command from within the ECS task (if possible):

aws secretsmanager get-secret-value --secret-id myapp/staging/credentials --region your-region-code

Replace "your-region-code" with the appropriate AWS Region.

5. Review your application code to ensure it's using the correct secret name and Region.

6. Check the CloudWatch logs for your ECS task to see if there are any additional error messages that might provide more context.

If you've confirmed all these steps and are still experiencing issues, you may want to temporarily add logging in your application code to output more detailed error information about the Secrets Manager access attempt.

Remember, the Task Role is used for permissions that your application code needs at runtime, while the Task Execution Role is used by the ECS agent to pull container images and send logs to CloudWatch. In this case, since your application is trying to access Secrets Manager at runtime, the permissions need to be on the Task Role.  
**Sources**  
[Troubleshoot permissions issues in Amazon ECS | AWS re:Post](https://repost.aws/knowledge-center/ecs-troubleshoot-permissions)  
[Manage secrets and access keys for Amazon ECS | AWS re:Post](https://repost.aws/knowledge-center/ecs-manage-secrets-access-keys)

Aakash Sai Raj · Answer

Hello,

Based on the error message, I notice this is actually about Secrets Manager, not SSM Parameter Store. The error specifically mentions aws-secretsmanager:myapp/staging/credentials.

The Task Execution Role is used by ECS agent to pull container images, sensitive data using Secrets Manager secrets or AWS Systems Manager Parameter Store parameters. Please refer[1].

Therefore, you can try adding this policy to the task execution role.
`````````
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue"
            ],
            "Resource": ""
        }
    ]
}
`````````

Incase if you want to provide access to the ECS task with any parameter from  systems manager, then below is the policy that should be added. 
`````````
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameters"
            ],
            "Resource": ""
        }
    ]
}
`````````

Furthermore, ensure to review the CloudTrail events history for  GetSecretValue/GetParameters accordingly to identify any failed API calls.

You may also review this rePost article[2] for more information.

[1] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html

[2] https://repost.aws/knowledge-center/ecs-data-security-container-task

Amazon ECS container permissions issue

Relevant content