aws ecs execute-command fails with errors related to KMS and Cloudwatch

Question

Hello all.

I've deployed ECS Fargate services with enabled `execute-command` capabilities. When I execute `aws ecs execute-command` I'm getting following errors:

```
Encountered error while initiating handshake. Fetching data key failed: Unable to retrieve data key, Error when decrypting data key AccessDeniedException: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
```

This was fixed by adding IAM permissions

```
"Action": "kms:Decrypt",
"Effect": "Allow",
"Resource": "arn:aws:kms:us-east-2:1234567890:*",
"Sid": "AllowDecryptData"
```

After that, I get following error

```
We couldn't start the session because encryption is not set up on the selected CloudWatch Logs log group. Either encrypt the log group or choose an option to enable logging without encryption.
```

Weird thing is, in Session Manager Preferences all options are set to `off`. Enabling CloudWatch logging and disabling encryption does not change the behavior.

Task role has following permissions:

```
{
    "Effect": "Allow",
    "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
    ],
    "Resource": "*"
}
{
    "Sid": "",
    "Effect": "Allow",
    "Action": [
        "ssmmessages:OpenDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:CreateControlChannel"
    ],
    "Resource": "*"
}
{
    "Action": "kms:Decrypt",
    "Effect": "Allow",
    "Resource": "arn:aws:kms:us-east-2:1234567890:*",
    "Sid": "AllowDecryptData"
}
```

Additionally `AmazonSSMManagedInstanceCore` policy is attached to the role

Task Execution role has

```
{
    "Sid": "",
    "Effect": "Allow",
    "Action": [
        "logs:PutLogEvents",
        "logs:CreateLogStream",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetAuthorizationToken",
        "ecr:BatchGetImage",
        "ecr:BatchCheckLayerAvailability"
    ],
    "Resource": "*"
}
```

Additionally `AmazonSSMManagedInstanceCore` policy is attached to the role

What am I missing?

Thank you.

Accepted Answer

Have you associated your log group with a customer managed key (CMK)? Please see [Encrypt log data in CloudWatch Logs using AWS Key Management Service](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/encrypt-log-data-kms.html). If you set the `cloudWatchEncryptionEnabled` value as true while creating a cluster, that setting expects the log group to be encrypted with a CMK (KMS key). If you don't need to use that option, you can call[ `UpdateCluster`](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateCluster.html) to set `cloudWatchEncryptionEnabled` as false. Even if you set it as false, `Log group data is always encrypted in CloudWatch Logs`.

aws ecs execute-command fails with errors related to KMS and Cloudwatch

Relevant content