EMR Studio creation error: The service role does not have permission to access the <CLUSTER>

0

Hi,

trying to attach a emr studio and workspace to a emr cluster via terraform. But get an error saying:

 Error: creating EMR Studio: InvalidRequestException: The service role does not have permission to access the dev-emr. Review the policies for the service role and the S3 bucket policy to ensure that the permission is granted. (Service: AWSEditors; Status Code: 400; Error Code: InvalidRequestException; Request ID: 9c904871-78e2-4893-88a4-f6410b3da4b0; Proxy: null)
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "9d6bd37a-c61f-4ad2-87e8-8eb9c4cb0f27"
│   },
│   Message_: "The service role does not have permission to access the dev-emr. Review the policies for the service role and the S3 bucket policy to ensure that the permission is granted. (Service: AWSEditors; Status Code: 400; Error Code: InvalidRequestException; Request ID: 9c904871-78e2-4893-88a4-f6410b3da4b0; Proxy: null)"
│ }
│ 
│   with aws_emr_studio.emr-studio,
│   on emr-studio.tf line 1, in resource "aws_emr_studio" "emr-studio":
│    1: resource "aws_emr_studio" "emr-studio" {

Studio code is:

resource "aws_emr_studio" "emr-studio" {
  name                        = "${local.full_name}-studio"
  auth_mode                   = "IAM"
  default_s3_location         = "s3://${var.s3_prevent_destroy == true ? aws_s3_bucket.emr[0].id : aws_s3_bucket.emr-destroy[0].id}/studio"
  engine_security_group_id    = aws_security_group.emr-studio.id
  service_role                = aws_iam_role.iam_emr_service_role.arn
  subnet_ids                  = [data.aws_subnets.intra.ids[0], data.aws_subnets.intra.ids[1], data.aws_subnets.intra.ids[2]]
  #user_role                   = aws_iam_role.iam_studio_user_role.arn
  vpc_id                      = data.aws_vpc.this.id
  workspace_security_group_id = aws_security_group.emr-studio.id
}

My iam role has

resource "aws_iam_role" "studio_service_role" {
  name = "${local.full_name}-studio-service-role"
  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Principal" : {
          "Service" : "elasticmapreduce.amazonaws.com"
        },
        "Action" : "sts:AssumeRole",
    }]
  })
}

and the following policy attached:

resource "aws_iam_policy" "studio_service_role" {
  name        = "${local.full_name}-studio-service-policy"
  description = "copy of EMR_Notebooks_DefaultRole policy with extras"
  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
       {
      "Sid": "AllowEMRReadOnlyActions",
      "Effect": "Allow",
      "Action": [
        "elasticmapreduce:ListInstances",
        "elasticmapreduce:DescribeCluster",
        "elasticmapreduce:ListSteps"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowEC2ENIActionsWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterfacePermission",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:network-interface/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/for-use-with-amazon-emr-managed-policies": "true"
        }
      }
    },
    {
      "Sid": "AllowEC2ENIAttributeAction",
      "Effect": "Allow",
      "Action": [
        "ec2:ModifyNetworkInterfaceAttribute"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:network-interface/*",
        "arn:aws:ec2:*:*:security-group/*"
      ]
    },
    {
      "Sid": "AllowEC2SecurityGroupActionsWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:DeleteNetworkInterfacePermission"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/for-use-with-amazon-emr-managed-policies": "true"
        }
      }
    },
    {
      "Sid": "AllowDefaultEC2SecurityGroupsCreationWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateSecurityGroup"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:security-group/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/for-use-with-amazon-emr-managed-policies": "true"
        }
      }
    },
    {
      "Sid": "AllowDefaultEC2SecurityGroupsCreationInVPCWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateSecurityGroup"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:vpc/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/for-use-with-amazon-emr-managed-policies": "true"
        }
      }
    },
    {
      "Sid": "AllowAddingEMRTagsDuringDefaultSecurityGroupCreation",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateTags"
      ],
      "Resource": "arn:aws:ec2:*:*:security-group/*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/for-use-with-amazon-emr-managed-policies": "true",
          "ec2:CreateAction": "CreateSecurityGroup"
        }
      }
    },
    {
      "Sid": "AllowEC2ENICreationWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:network-interface/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/for-use-with-amazon-emr-managed-policies": "true"
        }
      }
    },
    {
      "Sid": "AllowEC2ENICreationInSubnetAndSecurityGroupWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:subnet/*",
        "arn:aws:ec2:*:*:security-group/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/for-use-with-amazon-emr-managed-policies": "true"
        }
      }
    },
    {
      "Sid": "AllowAddingTagsDuringEC2ENICreation",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateTags"
      ],
      "Resource": "arn:aws:ec2:*:*:network-interface/*",
      "Condition": {
        "StringEquals": {
          "ec2:CreateAction": "CreateNetworkInterface"
        }
      }
    },
    {
      "Sid": "AllowEC2ReadOnlyActions",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeTags",
        "ec2:DescribeInstances",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowSecretsManagerReadOnlyActionsWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:*:*:secret:*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/for-use-with-amazon-emr-managed-policies": "true"
        }
      }
    },
    {
      "Sid": "AllowWorkspaceCollaboration",
      "Effect": "Allow",
      "Action": [
        "iam:GetUser",
        "iam:GetRole",
        "iam:ListUsers",
        "iam:ListRoles",
        "sso:GetManagedApplicationInstance",
        "sso-directory:SearchUsers"
      ],
      "Resource": "*"
      },
      {
        "Effect" : "Allow",
        "Action" : "secretsmanager:GetSecretValue",
        "Resource" : "*"
      },
      {
        "Effect" : "Allow",
        "Action" : "s3:*",
        "Resource" : "*"
      },
      {
        "Effect" : "Allow",
        "Action" : [
          "ec2:AuthorizeSecurityGroupEgress",
          "ec2:AuthorizeSecurityGroupIngress",
          "ec2:CreateSecurityGroup",
          "ec2:DescribeSecurityGroups",
          "ec2:RevokeSecurityGroupEgress",
          "ec2:CreateNetworkInterface",
          "ec2:CreateNetworkInterfacePermission",
          "ec2:DeleteNetworkInterface",
          "ec2:DeleteNetworkInterfacePermission",
          "ec2:DescribeNetworkInterfaces",
          "ec2:ModifyNetworkInterfaceAttribute",
          "ec2:DescribeTags",
          "ec2:DescribeInstances",
          "ec2:DescribeSubnets",
          "ec2:DescribeVpcs",
          "elasticmapreduce:ListInstances",
          "elasticmapreduce:DescribeCluster",
          "elasticmapreduce:ListSteps"
        ],
        "Resource" : "*"
      },
      {
        "Effect" : "Allow",
        "Action" : "ec2:CreateTags",
        "Resource" : "arn:aws:ec2:*:*:network-interface/*",
        "Condition" : {
          "ForAllValues:StringEquals" : {
            "aws:TagKeys" : [
              "aws:elasticmapreduce:editor-id",
              "aws:elasticmapreduce:job-flow-id"
            ]
          }
        }
      }
    ]
  })
}

this is basically a copy and paste of all resource options from the following docs:

https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-service-role.html

https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-service-role.html

Any idea what im doing wrong?

asked a year ago1748 views
5 Answers
2

In your Terraform configuration, after creating the IAM role and policy, you also need to create an IAM role policy attachment. This resource links the policy to the role. Here's how you might do it:

resource "aws_iam_role_policy_attachment" "studio_service_role_attachment" {
  role       = aws_iam_role.studio_service_role.name
  policy_arn = aws_iam_policy.studio_service_role.arn
}

In your Terraform configuration, after creating the IAM role and policy, you also need to create an IAM role policy attachment. This resource links the policy to the role. Here's how you might do it:

resource "aws_iam_role_policy_attachment" "studio_service_role_attachment" {
  role       = aws_iam_role.studio_service_role.name
  policy_arn = aws_iam_policy.studio_service_role.arn
}
profile picture
EXPERT
answered a year ago
0

Hi, @sdtslmn

Sorry yes i have attached the policy to the role, I also found that i have to remove all the tagging logic as I wont be adding tags to all the resources needed to match that pattern.

My iam-studio-service-role.tf now looks like below:

# More info for IAM can be found:
# https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-service-role.html
# https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-service-role.html

resource "aws_iam_role" "studio_service_role" {
  name = "${local.full_name}-studio-service-role"
  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Principal" : {
          "Service" : "elasticmapreduce.amazonaws.com"
        },
        "Action" : "sts:AssumeRole",
    }]
  })
}

resource "aws_iam_role_policy_attachment" "studio_service_role" {
  role       = aws_iam_role.studio_service_role.name
  policy_arn = aws_iam_policy.studio_service_role.arn
}

resource "aws_iam_policy" "studio_service_role" {
  name        = "${local.full_name}-studio-service-policy"
  description = "copy of EMR_Notebooks_DefaultRole policy with extras"
  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
       {
      "Sid": "AllowEMRReadOnlyActions",
      "Effect": "Allow",
      "Action": [
        "elasticmapreduce:ListInstances",
        "elasticmapreduce:DescribeCluster",
        "elasticmapreduce:ListSteps"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowEC2ENIActionsWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterfacePermission",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:network-interface/*"
      ],
    },
    {
      "Sid": "AllowEC2ENIAttributeAction",
      "Effect": "Allow",
      "Action": [
        "ec2:ModifyNetworkInterfaceAttribute"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:network-interface/*",
        "arn:aws:ec2:*:*:security-group/*"
      ]
    },
    {
      "Sid": "AllowEC2SecurityGroupActionsWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:DeleteNetworkInterfacePermission"
      ],
      "Resource": "*",
    },
    {
      "Sid": "AllowDefaultEC2SecurityGroupsCreationWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateSecurityGroup"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:security-group/*"
      ],
    },
    {
      "Sid": "AllowDefaultEC2SecurityGroupsCreationInVPCWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateSecurityGroup"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:vpc/*"
      ],
    },
    {
      "Sid": "AllowAddingEMRTagsDuringDefaultSecurityGroupCreation",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateTags"
      ],
      "Resource": "arn:aws:ec2:*:*:security-group/*",
    },
    {
      "Sid": "AllowEC2ENICreationWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:network-interface/*"
      ],
    },
    {
      "Sid": "AllowEC2ENICreationInSubnetAndSecurityGroupWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:subnet/*",
        "arn:aws:ec2:*:*:security-group/*"
      ],
    },
    {
      "Sid": "AllowAddingTagsDuringEC2ENICreation",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateTags"
      ],
      "Resource": "arn:aws:ec2:*:*:network-interface/*",
    },
    {
      "Sid": "AllowEC2ReadOnlyActions",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeTags",
        "ec2:DescribeInstances",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowSecretsManagerReadOnlyActionsWithEMRTags",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:*:*:secret:*",
    },
    {
      "Sid": "AllowWorkspaceCollaboration",
      "Effect": "Allow",
      "Action": [
        "iam:GetUser",
        "iam:GetRole",
        "iam:ListUsers",
        "iam:ListRoles",
        "sso:GetManagedApplicationInstance",
        "sso-directory:SearchUsers"
      ],
      "Resource": "*"
      },
      {
        "Effect" : "Allow",
        "Action" : "secretsmanager:GetSecretValue",
        "Resource" : "*"
      },
      {
        "Effect" : "Allow",
        "Action" : "s3:*",
        "Resource" : "*"
      },
      {
        "Effect" : "Allow",
        "Action" : [
          "ec2:AuthorizeSecurityGroupEgress",
          "ec2:AuthorizeSecurityGroupIngress",
          "ec2:CreateSecurityGroup",
          "ec2:DescribeSecurityGroups",
          "ec2:RevokeSecurityGroupEgress",
          "ec2:CreateNetworkInterface",
          "ec2:CreateNetworkInterfacePermission",
          "ec2:DeleteNetworkInterface",
          "ec2:DeleteNetworkInterfacePermission",
          "ec2:DescribeNetworkInterfaces",
          "ec2:ModifyNetworkInterfaceAttribute",
          "ec2:DescribeTags",
          "ec2:DescribeInstances",
          "ec2:DescribeSubnets",
          "ec2:DescribeVpcs",
          "elasticmapreduce:ListInstances",
          "elasticmapreduce:DescribeCluster",
          "elasticmapreduce:ListSteps"
        ],
        "Resource" : "*"
      },
      {
        "Effect" : "Allow",
        "Action" : "ec2:CreateTags",
        "Resource" : "arn:aws:ec2:*:*:network-interface/*",
      }
    ]
  })
}

And i still get the error:

Error: creating EMR Studio: InvalidRequestException: The service role does not have permission to access the oc-dev-emr. Review the policies for the service role and the S3 bucket policy to ensure that the permission is granted. (Service: AWSEditors; Status Code: 400; Error Code: InvalidRequestException; Request ID: 38f2aa57-7ba0-48e0-899d-15c13db8c4f1; Proxy: null)
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "c8a17a22-609a-41b8-ac86-59950e786dbe"
│   },
│   Message_: "The service role does not have permission to access the oc-dev-emr. Review the policies for the service role and the S3 bucket policy to ensure that the permission is granted. (Service: AWSEditors; Status Code: 400; Error Code: InvalidRequestException; Request ID: 38f2aa57-7ba0-48e0-899d-15c13db8c4f1; Proxy: null)"
│ }
│ 
│   with aws_emr_studio.emr-studio,
│   on emr-studio.tf line 1, in resource "aws_emr_studio" "emr-studio":
│    1: resource "aws_emr_studio" "emr-studio" {
│ 

To add to my confusion! if i create a studio via the console using the same IAM role, it works successufully and lets bring up a workspace

answered a year ago
0

I am also having the same issues, any further update/resolution on this please? Thanks Suddhasil

answered 8 months ago
0

for me below policy worked :

resource "aws_iam_policy" "emr_studio_policy" {
  name        = "emr_studio_policy"
  description = "Allow all accesses needed for EMR"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
          {
          "Effect": "Allow",
          "Action": [
            "elasticmapreduce:*",
            "ec2:CreateNetworkInterfacePermission",
            "ec2:DeleteNetworkInterface"
          ],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": "elasticmapreduce:StartNotebookExecution",
          "Resource": "arn:aws:elasticmapreduce:us-east-1:account_no:notebook-instance/*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetObject",
            "s3:PutObject"
          ],
          "Resource": [
            local.bucket_arn_with_wildcard
          ]
        },
        {
          "Effect": "Allow",
          "Action": "s3:ListBucket",
          "Resource": var.s3_bucket_arn
        }
    ]
  })
}

this is the policy code, and then had to attach this policy to the role created.

Chetan
answered 7 months ago
0

This is my IAM role I succeed for EMR Studio. Hope it helps!

resource "aws_iam_role" "hm_amazon_emr_studio_iam_role" {
  name = "AmazonEMRStudioServiceRole-${var.amazon_emr_studio_name}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = "sts:AssumeRole"
        Principal = {
          Service = "elasticmapreduce.amazonaws.com"
        }
      }
    ]
  })
  tags = {
    Environment = var.environment
    Team        = var.team
    Name        = "AmazonEMRStudioServiceRole-${var.amazon_emr_studio_name}"
  }
}
resource "aws_iam_role_policy" "hm_amazon_emr_studio_iam_role_input_policy" {
  name = "AmazonEMRStudioServicePolicyForS3-${var.amazon_emr_studio_name}"
  role = aws_iam_role.hm_amazon_emr_studio_iam_role.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:DeleteObject",
          "s3:GetEncryptionConfiguration",
          "s3:GetObject",
          "s3:ListBucket",
          "s3:PutObject"
        ]
        Resource = [
          "arn:aws:s3:::${var.s3_bucket}",
          "arn:aws:s3:::${var.s3_bucket}/*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "s3:ListAllMyBuckets"
        ]
        Resource = [
          "*"
        ]
      }
    ]
  })
}
resource "aws_iam_role_policy_attachment" "hm_amazon_emr_studio_iam_role_policy_attachment" {
  role       = aws_iam_role.hm_amazon_emr_studio_iam_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceEditorsRole"
}
profile picture
answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions