By using AWS re:Post, you agree to the Terms of Use

aws ecs execute-command fails with errors related to KMS and Cloudwatch

0

Hello all.

I've deployed ECS Fargate services with enabled execute-command capabilities. When I execute aws ecs execute-command I'm getting following errors:

Encountered error while initiating handshake. Fetching data key failed: Unable to retrieve data key, Error when decrypting data key AccessDeniedException: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

This was fixed by adding IAM permissions

"Action": "kms:Decrypt",
"Effect": "Allow",
"Resource": "arn:aws:kms:us-east-2:1234567890:*",
"Sid": "AllowDecryptData"

After that, I get following error

We couldn't start the session because encryption is not set up on the selected CloudWatch Logs log group. Either encrypt the log group or choose an option to enable logging without encryption.

Weird thing is, in Session Manager Preferences all options are set to off. Enabling CloudWatch logging and disabling encryption does not change the behavior.

Task role has following permissions:

{
    "Effect": "Allow",
    "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
    ],
    "Resource": "*"
}
{
    "Sid": "",
    "Effect": "Allow",
    "Action": [
        "ssmmessages:OpenDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:CreateControlChannel"
    ],
    "Resource": "*"
}
{
    "Action": "kms:Decrypt",
    "Effect": "Allow",
    "Resource": "arn:aws:kms:us-east-2:1234567890:*",
    "Sid": "AllowDecryptData"
}

Additionally AmazonSSMManagedInstanceCore policy is attached to the role

Task Execution role has

{
    "Sid": "",
    "Effect": "Allow",
    "Action": [
        "logs:PutLogEvents",
        "logs:CreateLogStream",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetAuthorizationToken",
        "ecr:BatchGetImage",
        "ecr:BatchCheckLayerAvailability"
    ],
    "Resource": "*"
}

Additionally AmazonSSMManagedInstanceCore policy is attached to the role

What am I missing?

Thank you.

1 Answers
0
Accepted Answer

Have you associated your log group with a customer managed key (CMK)? Please see Encrypt log data in CloudWatch Logs using AWS Key Management Service. If you set the cloudWatchEncryptionEnabled value as true while creating a cluster, that setting expects the log group to be encrypted with a CMK (KMS key). If you don't need to use that option, you can call UpdateCluster to set cloudWatchEncryptionEnabled as false. Even if you set it as false, Log group data is always encrypted in CloudWatch Logs.

answered 18 days ago
  • Hey. Yes, cloudWatchEncryptionEnabled was set to true. After changing it to false, I am able to connect to container. Thank you.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions