ECS Fargate tasks stuck in stop and start cycle during deployment

Question

I am doing a deployment in ECS Fargate Service, I see the tasks are getting started successfully but then they are repeatedly getting stopped with "AWSSecurityTokenServiceException" for the task role. and new tasks are getting started. 
During testing on dev-desk, whenever this exception occurs, we update the test account credentials using ada command. But not sure, why tasks are failing in Beta stage with the same.

The Task Role has been given the relevant permissions. 
Trust policy of Task Role

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::339712882588:root"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
```

**Error Details :**

```
17:59:14.531 [main] WARN  org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'tomcatServletWebServerFactory' defined in class path resource [org/springframework/boot/autoconfigure/web/servlet/ServletWebServerFactoryConfiguration$EmbeddedTomcat.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'containerCustomizer' defined in class path resource [com/amazon/embed/tomcat/autoconfigure/EmbeddedTomcatConfiguration.class]: Unsatisfied dependency expressed through method 'containerCustomizer' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'queryLogValve' defined in class path resource [com/amazon/embed/tomcat/autoconfigure/EmbeddedTomcatConfiguration.class]: Unsatisfied dependency expressed through method 'queryLogValve' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'metricsFactory' defined in class path resource [com/amazon/metrics/autoconfigure/MetricsConfiguration.class]: Unsatisfied dependency expressed through method 'metricsFactory' parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cloudWatchReporterFactory' defined in class path resource [com/amazon/metrics/autoconfigure/CloudWatchMetricsConfiguration.class]: Unsatisfied dependency expressed through method 'cloudWatchReporterFactory' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cloudWatchAmazonCloudWatchAsync' defined in class path resource [com/amazon/metrics/autoconfigure/CloudWatchMetricsConfiguration.class]: Unsatisfied dependency expressed through method 'cloudWatchAmazonCloudWatchAsync' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'awsCredentialsProvider' defined in class path resource [com/rewardsmarketinghubservicecore/module/RewardsMarketinghubServiceCoreModule.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.amazonaws.auth.AWSCredentialsProvider]: Factory method 'getAwsCredentialsProvider' threw exception; nested exception is java.lang.RuntimeException: Failed to assume role: User: arn:aws:sts::339712882588:assumed-role/EcsService-Alpha-EcsTaskInstanceRoleE38DB492-rq6N7a8Mxax9/0a7847a0c3c14015be3ff1deef081be1 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::339712882588:role/EcsService-Alpha-EcsTaskInstanceRoleE38DB492-rq6N7a8Mxax9 (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: 2baae3cd-6b45-4c01-8c2e-d03ba9843d55; Proxy: null)
.
.
.
Caused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: User: arn:aws:sts::339712882588:assumed-role/EcsService-Alpha-EcsTaskInstanceRoleE38DB492-rq6N7a8Mxax9/0a7847a0c3c14015be3ff1deef081be1 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::339712882588:role/EcsService-Alpha-EcsTaskInstanceRoleE38DB492-rq6N7a8Mxax9 (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: 2baae3cd-6b45-4c01-8c2e-d03ba9843d55; Proxy: null)
```

rePost-User-7973184 · Accepted Answer

Using DefaultAWSCredsProvider Chain to use the creds for the Task Role.

@Bean(name = "awsCredentialsProvider")
    public AWSCredentialsProvider getAwsCredentialsProvider() {
        return DefaultAWSCredentialsProviderChain.getInstance();
    }

Jérôme GUYON · Answer

Hello, 
From your logs : 
> Failed to assume role: User: arn:aws:sts::339712882588:assumed-role/EcsService-Alpha-EcsTaskInstanceRoleE38DB492-rq6N7a8Mxax9/0a7847a0c3c14015be3ff1deef081be1 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::339712882588:role/EcsService-Alpha-EcsTaskInstanceRoleE38DB492-rq6N7a8Mxax9

It look likes your task, already assuming the role, "EcsService-Alpha-EcsTaskInstanceRoleE38DB492-rq6N7a8Mxax9" is trying to assume it again.

If you really need the role to assume itself please :

#1. Ensure you have an IAM Policy allowing Assuming the role itself. You can add the following inline policy to the role : 
```
{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "sts:AssumeRole"
         ],
         "Resource":[
            "arn:aws:iam::339712882588:role/EcsService-Alpha-EcsTaskInstanceRoleE38DB492-rq6N7a8Mxax9"
         ]
      }
   ]
}
```

#2.  In the trust policy, use least-privilege on role instead of allowing to assume from the whole account "arn:aws:iam::339712882588:root"
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::339712882588:role/EcsService-Alpha-EcsTaskInstanceRoleE38DB492-rq6N7a8Mxax9"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
``` 
#3.
Additionally you can find here  the documentation about Task Roles : https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

PS : Instead of self-assuming roles, you may consider having distinct roles with least-privilege strategy for your ECS task configurations

ECS Fargate tasks stuck in stop and start cycle during deployment

Add your answer

Relevant content