ECS Exec error TargetNotConnectedException

Question

I'm getting this error when I try to connect to my Fargate ECS container using ECS Exec feature:

```
An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.
```
The task's role was missing the SSM permissions, I added them, but the error persists.
Below is the output of the [amazon-ecs-exec-checker](https://github.com/aws-containers/amazon-ecs-exec-checker) tool:

```
Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.2.1 Python/3.8.8 Linux/5.13.0-39-generic exe/x86_64.ubuntu.20 prompt/off)
  Session Manager Plugin | OK (1.2.295.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : us-west-2
Cluster: removed
Task   :  removed
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::removed
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Container-Level Checks | 
    ----------
      Managed Agent Status
    ----------
         1. RUNNING for "api-staging"
    ----------
      Init Process Enabled (api-staging:14)
    ----------
         1. Enabled - "api-staging"
    ----------
      Read-Only Root Filesystem (api-staging:14)
    ----------
         1. Disabled - "api-staging"
  Task Role Permissions  | arn:aws:iam::removed
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
  VPC Endpoints          | 
    Found existing endpoints for vpc-removed:
      - com.amazonaws.us-west-2.s3
    SSM PrivateLink "com.amazonaws.us-west-2.ssmmessages" not found. You must ensure your task has proper outbound internet connectivity.  Environment Variables  | (api-staging:14)
       1. container "api-staging"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: defined
       - AWS_SECRET_ACCESS_KEY: defined
```
The output has only green and yellow markers, no red ones, which means Exec should work. But it doesn't. Any ideas why?

Answer

Ah, I think I know why it doesn't work. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars are defined and it's likely they don't have the SSM permissions. It looks like Exec will use the env vars instead of the IAM role. I'll update my post a bit later when I fix the env vars issue.

ECS Exec error TargetNotConnectedException

Relevant questions

MaxInstanceLifetime value for ECS Fargate

Exec format error on Elastic Beanstalk postdeploy script

Can't create ECS deployment group using CloudFormation (loadBalancerInfo error)

No space left on device when deploy ECS Fargate Service with CDK

Unable to override taskRoleArn when running ECS task from Lambda

ECS Exec error TargetNotConnectedException

ECS Fargate Task Logs Empty When Using a Entrypoint Shell Script

Aws ecs task failed ELB health check

exec format error trying to use 64-bit ARM architecture for Fargate launch type

Permission denied for "mkdir /usr/local/sessionmanagerplugin/logs/"