ECS Exec error TargetNotConnectedException

1

I'm getting this error when I try to connect to my Fargate ECS container using ECS Exec feature:

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

The task's role was missing the SSM permissions, I added them, but the error persists. Below is the output of the amazon-ecs-exec-checker tool:

Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.2.1 Python/3.8.8 Linux/5.13.0-39-generic exe/x86_64.ubuntu.20 prompt/off)
  Session Manager Plugin | OK (1.2.295.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : us-west-2
Cluster: removed
Task   :  removed
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::removed
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Container-Level Checks | 
    ----------
      Managed Agent Status
    ----------
         1. RUNNING for "api-staging"
    ----------
      Init Process Enabled (api-staging:14)
    ----------
         1. Enabled - "api-staging"
    ----------
      Read-Only Root Filesystem (api-staging:14)
    ----------
         1. Disabled - "api-staging"
  Task Role Permissions  | arn:aws:iam::removed
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
  VPC Endpoints          | 
    Found existing endpoints for vpc-removed:
      - com.amazonaws.us-west-2.s3
    SSM PrivateLink "com.amazonaws.us-west-2.ssmmessages" not found. You must ensure your task has proper outbound internet connectivity.  Environment Variables  | (api-staging:14)
       1. container "api-staging"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: defined
       - AWS_SECRET_ACCESS_KEY: defined

The output has only green and yellow markers, no red ones, which means Exec should work. But it doesn't. Any ideas why?

asked 2 years ago2521 views
3 Answers
1

Ah, I think I know why it doesn't work. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars are defined and it's likely they don't have the SSM permissions. It looks like Exec will use the env vars instead of the IAM role. I'll update my post a bit later when I fix the env vars issue.

answered 2 years ago
  • Yes, adding SSM permissions to the user behind AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars fixed the problem for me.

  • I had this issue also but had to add ecs:ExecuteCommand permissions to the AWS_ACCESS_KEY_ID account in addition to the SSM permissions.

  • Granting SSM permissions to the user account associated with the env var KEY_ID worked for me. Was something changed recently in Fargate so that this credential controls over the IAM role of the task? About a month ago, I was able to connect without SSM permissions attached to the internally-referenced user account.

0

Hello, I know this discussion was about a year ago in 2022, but I am faced with same/similar problem today. I am getting same exact error trying to execute command on a simple bitnami/nginx container running on ECS EC profile:

aws ecs execute-command --task <removed> --cluster awsome_ecs_cluster --container nginx-test1 --interactive --command "/bin/bash" The Session Manager plugin was installed successfully. Use the AWS CLI to start a session. An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

Running the https://github.com/aws-containers/amazon-ecs-exec-checker tool everything appears to be running OK:

Prerequisites for check-ecs-exec.sh v0.7

jq | OK (/opt/homebrew/bin/jq) AWS CLI | OK (/usr/local/bin/aws)

Prerequisites for the AWS CLI to use ECS Exec

AWS CLI Version | OK (aws-cli/2.11.0 Python/3.11.2 Darwin/22.3.0 exe/x86_64 prompt/off) Session Manager Plugin | OK (1.2.398.0)

Checks on ECS task and other resources

Region : us-west-1 Cluster: awsome_ecs_cluster Task : removed

Cluster Configuration | KMS Key : Not Configured Audit Logging : DEFAULT S3 Bucket Name: Not Configured CW Log Group : Not Configured Can I ExecuteCommand? | arn:aws:iam::XXXXXXXXXX:user/dz-XXXXXXXXXXXX ecs:ExecuteCommand: allowed ssm:StartSession denied?: allowed Task Status | RUNNING Launch Type | EC2 ECS Agent Version | 1.68.0 Exec Enabled for Task | OK Container-Level Checks | ---------- Managed Agent Status ---------- 1. RUNNING for "nginx-test1" ---------- Init Process Enabled (run_nginx_ecs:7) ---------- 1. Enabled - "nginx-test1" ---------- Read-Only Root Filesystem (run_nginx_ecs:7) ---------- 1. Disabled - "nginx-test1" Task Role Permissions | arn:aws:iam::XXXXXXXXXXX:role/ECSTaskRoleAB3 ssmmessages:CreateControlChannel: allowed ssmmessages:CreateDataChannel: allowed ssmmessages:OpenControlChannel: allowed ssmmessages:OpenDataChannel: allowed VPC Endpoints | SKIPPED (vpc-XXXXXXXXXXXXXXXXXXX - No additional VPC endpoints required) Environment Variables | (run_nginx_ecs:7) 1. container "nginx-test1" - AWS_ACCESS_KEY: not defined - AWS_ACCESS_KEY_ID: not defined - AWS_SECRET_ACCESS_KEY: not defined

So there are no AWS_ACCESS_KEY defined in container, only in CLI context but I am still getting an error above, same as in the original post.

any further thoughts on troubleshooting from the community are appreciated! thanks, Daniel

profile picture
answered a year ago
0

I'm getting the same error.

answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions