Not able to bring up EMR EC2 instance

0

I am trying to create a EMR cluster with version 6.13.0 and spark installed on it with version Spark 3.4.1 But the service role attached gives error that it does not have EC2 permissions. I tried with providing full EC2 access to the service role as well.

I than used the create service role option which created a new service role with default VPC , subnet and security group. But still i get the same validation error. The default role had below policies attached.

  1. AmazonEMRServicePolicy_v2
  2. AmazonEMR-ServiceRole-Policy-20231116T130134

I manually attached "AmazonEC2FullAccess" as well but the error still comes.

"Service role arn:aws:iam: xxxxx/role/service-role/AmazonEMR-ServiceRole-20231116T130134 has insufficient EC2 permissions"

Kindly suggest what I am doing wrong.

3 Answers
4

Hi, As you have seen it, when choosing the “create service role option”, Amazon EMR automatically attaches two mentioned policies by default, encompassing permissions for EC2, CloudWatch, IAM, and more. However, encountering this error message might be from conflicting policies or overrides within your IAM settings for roles or users. To diagnose this issue, you can leverage, AWS CloudTrail logs to trace API calls directed to EMR. By analyzing these logs, you can pinpoint denied actions and uncover the root cause of conflicting permissions. Ensure that the role attached to your user account has sufficient permissions to create clusters and EC2 instances. Additionally, confirm the selection of the correct EC2 instance profile. This profile, a specific type of service role, is assigned to every EC2 instance in an Amazon EMR cluster upon when it is created. Ensuring the appropriate configuration of both user roles and instance profiles can help troubleshoot permission-related issues encountered. For more information please refer EC2 instance profile and Service role for Amazon EMR links

profile pictureAWS
BezuW
answered 7 months ago
  • Hi Bezu,

    Thank you for your response and I tried with a custom policy but i get the same error message. I checked the cloudtrail and found below errorMessage.

    "errorMessage": "You are not authorized to perform this operation. User: arn:aws:sts::xxxxxxx:assumed-role/dce20-emr-dev-service-datarecon/CCSSession is not authorized to perform: ec2:RunInstances on resource: arn:aws:ec2:eu-west-1:xxxxx:volume/* with an explicit deny in a service control policy. Encoded authorization failure message: LmTxWvrwJxyAI84g8qo3B2BE9bSymkjqur4duvdX0Na9ZhKR3rUQ24zGEc5B30QqU1EIGzxalzgKsEJZr5gXLKD96MQZEBWpdsYVGGMvL2zgYI6jYKMrSK-B6ck66PfIqKrCl_UaGBzEjgIdz4NiRRC0JjZAfcGwm-l_1Cw1aAdctlkmqxwfA0B3gmZF3_Dlc1RrQUL0DLsgio8UM3sbyA1meJbC3NL2LHJnZhyi0CUnDphSgjmuJRMrOBcIHlCns245YbaQ2CLHrYZkoD6xbXdYW7dUJij9MyAV5ubO-0Y1shlBXUOLm5o8DCXu2p6S5GEWiYoD5G2-DRt_-WdIzHfMiu-DbyjedtliwArRDo2WEZhENCrIM63mzNMQpR4Sn_CwO7vXnQor7FDmRcRCTnouOJCieTLTEu1Qb9EEjVe2S0lQbnPFKTk1z7gQqdCB7rqCY6WSxZHVeY_VqpA8ftnvzN-4QcjIUQDSWGoh-OI2FcrFVZSF55yg6TCMU5itlAQ1cE1P-UF5CP14Nxfr9cH4uryvOVSJTcVU8BX3s5bH1GcAV_PlLNAmsJBNpBt48sIFUmpMhXj1MzcKjkwRPLtUR3ptW90ZorkadfHmG27hpgiXYfIztRenZ5YumNJxlmvY4vs4k_8aZH8vZq1jpVy7yArUqDyCqFnE7pavNlrSmX2Fn..."

  • Hi Amandeep,

    The "explicit deny" could be due to the SCPs permissions at org level that explicitly deny your action. Please note that "explicit deny" in SCP always overrides any allow in any other policy attached to the account. So check the SCP attached to your account (user or IAM role) and ensure it permits ec2:RunInstances. You can use this link https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_info-operations.html to check the policies attached to your account, organization. If you don't have sufficient permission check with your admin.

3

Hello,

If you have provided appropriate EC2 permissions, still the service role assume trust policy of elasticmapreduce.amazonaws.com. You can try using EMR_DefaultRole service role which is default role available to test if it is working or not. Alternatively you can create the policy with below json for an instance and then trust relationships should add the trust policy as shown below,

Example EMR role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Resource": "*",
            "Action": [
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CancelSpotInstanceRequests",
                "ec2:CreateFleet",
                "ec2:CreateLaunchTemplate",
                "ec2:CreateNetworkInterface",
                "ec2:CreateSecurityGroup",
                "ec2:CreateTags",
                "ec2:DeleteLaunchTemplate",
                "ec2:DeleteNetworkInterface",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteTags",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeImages",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeLaunchTemplates",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribePrefixLists",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSpotInstanceRequests",
                "ec2:DescribeSpotPriceHistory",
                "ec2:DescribeSubnets",
                "ec2:DescribeTags",
                "ec2:DescribeVpcAttribute",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeVpcEndpointServices",
                "ec2:DescribeVpcs",
                "ec2:DetachNetworkInterface",
                "ec2:ModifyImageAttribute",
                "ec2:ModifyInstanceAttribute",
                "ec2:RequestSpotInstances",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RunInstances",
                "ec2:TerminateInstances",
                "ec2:DeleteVolume",
                "ec2:DescribeVolumeStatus",
                "ec2:DescribeVolumes",
                "ec2:DetachVolume",
                "iam:GetRole",
                "iam:GetRolePolicy",
                "iam:ListInstanceProfiles",
                "iam:ListRolePolicies",
                "iam:PassRole",
                "s3:CreateBucket",
                "s3:Get*",
                "s3:List*",
                "sdb:BatchPutAttributes",
                "sdb:Select",
                "sqs:CreateQueue",
                "sqs:Delete*",
                "sqs:GetQueue*",
                "sqs:PurgeQueue",
                "sqs:ReceiveMessage",
                "cloudwatch:PutMetricAlarm",
                "cloudwatch:DescribeAlarms",
                "cloudwatch:DeleteAlarms",
                "application-autoscaling:RegisterScalableTarget",
                "application-autoscaling:DeregisterScalableTarget",
                "application-autoscaling:PutScalingPolicy",
                "application-autoscaling:DeleteScalingPolicy",
                "application-autoscaling:Describe*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot*",
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": "spot.amazonaws.com"
                }
            }
        }
    ]
}

Example Trust policy:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "elasticmapreduce.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

You can test the custom IAM policy using IAM Simulator. More information here - https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html

AWS
SUPPORT ENGINEER
answered 7 months ago
  • Hi Yokesh, Thank you for your inputs and I tried with a custom policy with the Json provided by you along with the trust policy but i get the same error message. I checked the cloudtrail and found below errorMessage.

    "errorMessage": "You are not authorized to perform this operation. User: arn:aws:sts::xxxxxxx:assumed-role/dce20-emr-dev-service-datarecon/CCSSession is not authorized to perform: ec2:RunInstances on resource: arn:aws:ec2:eu-west-1:xxxxx:volume/* with an explicit deny in a service control policy. Encoded authorization failure message: LmTxWvrwJxyAI84g8qo3B2BE9bSymkjqur4duvdX0Na9ZhKR3rUQ24zGEc5B30QqU1EIGzxalzgKsEJZr5gXLKD96MQZEBWpdsYVGGMvL2zgYI6jYKMrSK-B6ck66PfIqKrCl_UaGBzEjgIdz4NiRRC0JjZAfcGwm-l_1Cw1aAdctlkmqxwfA0B3gmZF3_Dlc1RrQUL0DLsgio8UM3sbyA1meJbC3NL2LHJnZhyi0CUnDphSgjmuJRMrOBcIHlCns245YbaQ2CLHrYZkoD6xbXdYW7dUJij9MyAV5ubO-0Y1shlBXUOLm5o8DCXu2p6S5GEWiYoD5G2-DRt_-WdIzHfMiu-DbyjedtliwArRDo2WEZhENCrIM63mzNMQpR4Sn_CwO7vXnQor7FDmRcRCTnouOJCieTLTEu1Qb9EEjVe2S0lQbnPFKTk1z7gQqdCB7rqCY6WSxZHVeY_VqpA8ftnvzN-4QcjIUQDSWGoh-OI2FcrFVZSF55yg6TCMU5itlAQ1cE1P-UF5CP14Nxfr9cH4uryvOVSJTcVU8BX3s5bH1GcAV_PlLNAmsJBNpBt48sIFUmpMhXj1MzcKjkwRPLtUR3ptW90ZorkadfHmG27hpgiXYfIztRenZ5YumNJxlmvY4vs4k_8aZH8vZq1jpVy7yArUqDyCqFnE7pavNlrSmX2Fn..."

  • Hi Amandeep,

    Seems like you have service control policy at org level deny to action ec2:RunInstances on resource like EBS volume *****:volume/ attachment.

    So, check if any SCP policy that denies like below, there might be some scenario, you are not allowed to create volume or instance without encryption or tag or any specific condition.

               "ec2:RunInstances",
                "ec2:CreateVolume"
    

    https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html

0

Amazon EMR uses an IAM service role to perform actions on your behalf to provision and manage clusters. The service role for cluster EC2 instances, also called the EC2 instance profile for Amazon EMR, is a special type of service role assigned to every EC2 instance in a cluster at launch. Define a custom EC2 instance profile to use instead of the EMR_EC2_DefaultRole when you launch your cluster.

profile pictureAWS
SUPPORT ENGINEER
Ravid_G
answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions