EMR 6.10 - local disk encryption error

Question

I'm trying to deploy Hadoop on EMR EC2 Cluster and having an issue with **local disk encryption error** on AWS Console.
AWS doesn't provide me any deeper logs nor info about the issue (S3 bucket log is empty, CloudTrail doesn't have Error Codes).

**My Security Configuration: use KMS CMK, Encryption Local disk, IMDSv2, AuthN: Off, AuthZ: Instance Profile.**

```
Used Roles:
EMR_DefaultRole (Policies : AmazonEMRServicePolicy_v2, myCustomPolicy)
EMR_EC2_DefaultRole (Policies : AmazonElasticMapReduceforEC2Role, myCustomPolicy)
```

myCustomPolicy:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "HadoopPolicy",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeSecurityGroups",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:CreateSecurityGroup",
                "ec2:DeleteSecurityGroup",
                "ec2:*"
            ],
            "Resource": [
                "arn:aws:ec2:my-region:XXXXXXXXXXX:security-group/sg-SLAVE",
                "arn:aws:ec2:my-region:XXXXXXXXXXX:security-group/sg-MASTER",
                "arn:aws:ec2:my-region:XXXXXXXXXXX:subnet/my-subnet-with-internet-access",
                "arn:aws:ec2:my-region:XXXXXXXXXXX:*"
            ]
        },
        {
            "Sid": "AllowReadAccesstoKey",
            "Effect": "Allow",
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey",
                "kms:CreateGrant",
                "kms:ListGrants"
            ],
            "Resource": "arn:aws:kms:my-region:XXXXXXXXXXX:key/my-kms-cmk-id"
        }
    ]
}
```

In addition:
* I'm using standard Amazon Linux AMI.
* In KMS CMK I pointed my EMR_DefaultRole and EMR_EC2_DefaultRole as a users and administrators of KEY.
* I've deployed cluster in subnet with Internet connection through IGW.

Reference guides:
1. https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-data-encryption-options.html
2. https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-security.html
3. https://repost.aws/knowledge-center/emr-create-cluster-with-ebs-encryption

Answer

Hi,
IAM permissions looks correctly. Instance profile which is used to spin EMR cluster instance has to be able to interact KMS key which will be used to encrypt EBS storage. You stated also that deployment was done in subnet with IGW. Can you verify if indeed IGW is in route table for that subnet?
What I would suggest to check is to make sure that VPC is properly configured. 
KMS service must be accessible from subnet(s) where cluster nodes are being created. Preferred way one can achieve that is by creating VPC Endpoint for KMS service (https://docs.aws.amazon.com/kms/latest/developerguide/kms-vpc-endpoint.html#vpce-create-endpoint). This way will make sure traffic will not traverse via public internet but stay inside AWS network.

Answer

Hello,

I see you also indicated how you setup the EMR security configuration. if no insights on Ctrail, on your S3 log directory go to the prefix /cluster_ID/node//setup-devices/ and refer to the log DiskEncryptor.log.gz for insights.

If you still not seeing any insights on WHY, please indicate will also test the same using your above steps

Answer

Hi, Thanks for your answers.

Finally, after many days I found the solution:
DELETE aws:SourceArn and aws:SourceAccount CONDITIONS from trust relationships for Instance Profile.

In my case when this conditions are in trust relationships lunching flow ends with "local disk encryption error - Internal error", logs are not publishing on S3 and there's no info in CloudTrail.

EMR_EC2_DefaultRole:

Trust relationships:

```
{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
/*START DELETE */
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": ""
                },
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:elasticmapreduce:::*"
                }
            }
/*END DELETE */
        }
    ]
}
```
I don't know why in AWS Documentation they provide this condition in Trust relationships:
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-iam-role.html

EMR 6.10 - local disk encryption error

相關內容