How do I troubleshoot Access denied errors for IAM users and roles in EMR Serverless?

Short description

The "Access denied" error in Amazon EMR Serverless is almost always caused by a misconfiguration on one of two roles. The caller identity is the user, role, AWS Lambda function, AWS Step Functions state machine, or Amazon Managed Workflows for Apache Airflow (Amazon MWAA) environment that calls the EMR Serverless API. The job runtime role, also called the execution role, is the role that EMR Serverless assumes to run your job. If you apply the fix to the wrong role, then you continue to receive the error.

This error occurs for one of the following reasons:

• The caller identity is missing the required emr-serverless actions in its identity-based policy.
• The caller identity is missing iam:PassRole permission to pass the job runtime role to EMR Serverless.
• The iam:PassRole statement uses the wrong iam:PassedToService condition or an execution-role ARN that doesn't match the StartJobRun call.
• A permissions boundary or an EMR Studio session policy restricts the emr-serverless actions.
• The job runtime role trust policy doesn't name emr-serverless.amazonaws.com as a trusted principal, or is missing sts:TagSession.
• An aws:RequestTag or aws:ResourceTag condition, an Amazon ECR repository policy, or an AWS Secrets Manager key policy blocks the action.

To resolve the error, identify which role and which condition apply to the error message that you receive, and then update that policy.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

The caller identity is misconfigured

You don't have the necessary permissions for EMR Serverless

When your IAM user or role doesn't have permissions to use EMR Serverless, you receive an error message that's similar to the following message:

"You do not have necessary permissions for EMR Serverless."

To resolve this error, grant your IAM user or role the permissions to use EMR Serverless.

The user isn't authorized to perform an action on a resource

When your IAM user or role doesn't have the necessary permissions to perform a specific action on a resource, you receive an error message similar to the following example:

"User: arn:aws:sts:example-region:example-aws-account:ZZZ/YYY is not authorized to perform: example-resource:action on resource: arn:aws:elasticmapreduce:example-region:example-AWS-account:example-resource/action because no identity-based policy allows the example-resource:action"

To resolve this error, identify the action that isn't allowed, and then add the action to your role's policy.

For example, in the following error message, the elasticmapreduce:CreateStudioPresignedUrl action isn't allowed:

"User: arn:aws:sts::example-AWS-account:assumed-role/example-role/example-user assuming the role is not authorized to perform: elasticmapreduce:CreateStudioPresignedUrl on resource: arn:aws:elasticmapreduce:example-region:example-AWS-account:studio/es-### because no identity-based policy allows the elasticmapreduce:CreateStudioPresignedUrl action"

Add all the actions your workflow uses, scoped to your application ARN in your role’s policy.

Example:

{
  "Effect": "Allow",
  "Action": [
    "emr-serverless:StartApplication",
    "emr-serverless:StopApplication",
    "emr-serverless:StartJobRun",
    "emr-serverless:GetJobRun",
    "emr-serverless:ListJobRuns"
  ],
  "Resource": "arn:aws:emr-serverless:region:account-id:/applications/application-id"
}

Note: Replace region with your AWS Region, account-id with your account ID, and application-id with your Application ID. Make sure you add all your workflows, including StartApplication, ListJobRuns, StartJobRun, and StopApplication.

The caller can't pass the execution role (iam:PassRole)

When you submit a job, you pass the execution role to EMR Serverless. If the caller is missing iam:PassRole, then you receive the following error:

"The caller receives AccessDeniedException on StartJobRun naming iam:PassRole on the execution-role ARN: ...is not authorized to perform: iam:PassRole on resource: arn:aws:iam::account-id:role/execution-role because no identity-based policy allows the iam:PassRole action. Add iam:PassRole with Resource set to that execution-role ARN."

To resolve this error, add iam:PassRole to the caller with Resource set to the execution role ARN.

If the condition restricts PassRole to the compute you call from, such as ec2.amazonaws.com then instead of the target service, the call is denied.

In the following example, the wrong iam:PassedToService condition is used:

{
  "Effect": "Allow",
  "Action": "iam:PassRole",
  "Resource": "arn:aws:iam:::role/execution-role-name",
  "Condition": {
    "StringEquals": {
      "iam:PassedToService": "emr-serverless.amazonaws.com"
    }
  }
}

To resolve this error, update the condition to emr-serverless.amazonaws.com.

The —execution-role-arn value in your StartJobRun call must match the ARN in the Resource field of the PassRole statement. If there’s a difference in account ID, path, or role name then you receive an access denied error.

To resolve this error, confirm that the —execution-role-arn value matches the ARN in the resource field.

Note: Check for AWS CloudFormation generated random suffixes on role names.

A permissions boundary is blocking EMR Serverless actions

When a role has a permissions boundary, the effective permissions are the intersection of the identity policy and the boundary. Even if the identity policy grants emr-serverless:CreateApplication, a boundary that doesn't list emr-serverless actions blocks it.

To resolve this error, add the required emr-serverless actions to the permissions boundary policy in addition to the identity policy.

An EMR Studio session policy is too restrictive

EMR Studio limits effective permissions to what the session policy explicitly allows, so two users on the same role can differ.

If you receive the "no session allows an action" error message, then the EMR Studio session policy that's attached to your user account is blocking that action.

To resolve this error, update the restrictive EMR Studio session policy assigned to the affected user to include the denied action.

StartJobRun is denied by unmet tag conditions

If the caller's policy allows emr-serverless:StartJobRun only under aws:RequestTag or aws:ResourceTag conditions, then the call is denied when it doesn't pass those tags.

To resolve this error, pass the exact tag keys named in the policy condition in the API request.

The job runtime role is misconfigured

The job runs but then fails to assume the execution role

If emr-serverless:StartJobRun succeeds but the job then fails with a message that the role "does not exist or is not set up with the required trust relationship," then EMR Serverless can't assume the runtime role at execution time.

To resolve this error, give the runtime role a trust policy that names the service principal.

Example:

{
  "Effect": "Allow",
  "Principal": {
    "Service": "emr-serverless.amazonaws.com"
  },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringEquals": {
      "aws:SourceAccount": "account-id"
    },
    "ArnLike": {
      "aws:SourceArn": ":/applications/ arn:aws:emr-serverless:region:account-id *"
    }
  }
}

Note: Replace region with your Region and account-id with your account ID.

If you reuse a role from another EMR service such as EMR Studio, then don't append emr-serverless.amazonaws.com to the existing statement. Its conditions won't match EMR Serverless ARNs. Add a separate statement, or create a dedicated role.

If a step function's definition references a role name but CloudFormation created the role with a generated suffix, then you get a "role-not-found or assumed" error. Make sure the referenced name matches exactly.

For Livy interactive endpoints, the role also needs emr-serverless:AccessLivyEndpoints in the identity policy and the trust relationship above.

Don't add principals that don't assume the runtime role such as airflow.amazonaws.com. MWAA calls the API but doesn't assume the runtime role.

The runtime role fails to assume itself (sts:TagSession)

If the runtime role is configured to assume itself but the trust policy is missing sts:TagSession, then you get “Access denied on sts:AssumeRole” error message. You might receive the error message even when sts:AssumeRole is present. EMR Serverless passes session tags during role assumption.

To resolve this error, add sts:TagSession to the trust policy actions.

Example:

{
  "Effect": "Allow",
  "Principal": {
    "Service": "emr-serverless.amazonaws.com"
  },
  "Action": [
    "sts:AssumeRole",
    "sts:TagSession"
  ]
}

A custom container image pull is denied (ecr:BatchGetImage)

If the ECR repository policy doesn't grant the service principal permission or the aws:SourceArn condition is wrong, then EMR Serverless can't pull a custom image from Amazon ECR.

To resolve this issue, add an ECR repository policy that grants the pull actions, scoped to your application ARN.

Example: 

{
  "Effect": "Allow",
  "Principal": {
    "Service": "emr-serverless.amazonaws.com"
  },
  "Action": [
    "ecr:BatchGetImage",
    "ecr:DescribeImages",
    "ecr:GetDownloadUrlForLayer"
  ],
  "Condition": {
    "StringEquals": {
      "aws:SourceArn": "arn:aws:emr-serverless:::/applications/*"
    }
  }
}

A Secrets Manager secret can't be read

EMR Serverless can't decrypt an AWS Secrets Manager secret that's encrypted with the default AWS Key Management Service (AWS KMS) key. It needs a customer managed key (CMK) with a key policy that grants the service principal decrypt access.

To resolve this error, re-encrypt the secret with a customer managed key and add the following updates to the key policy: 

{
  "Effect": "Allow",
  "Principal": {
    "Service": "emr-serverless.amazonaws.com"
  },
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "kms:ViaService": "secretsmanager.region.amazonaws.com"
    }
  }
}

It's a best practice to inject the secret into Spark using environment variables instead of an inline annotation inside spark.driver.extraJavaOptions. This keeps the secret out of process arguments and logs.

The issue isn't a caller or runtime-role permission

Amazon S3 access denied from inside a job

Amazon S3 access might be denied from inside a job when the wrong role has the Amazon S3 permission. Confirm whether the write is done by the runtime role or a separately assumed role. Then, grant s3:PutObject, s3:ListBucket, and related actions on the target bucket path to the role that performs the call.

Amazon S3 access might also be denied from inside a job when your S3 connector doesn't honor the assumed role configuration. If you use spark.hadoop.fs.s3a.assumed.role.arn (an S3A setting) to configure the assumed role, whether your other S3 writes honor that role depends on your EMR release.

On EMR release 7.10.0 and later, S3A is the default S3 connector for all schemes (s3://, s3n://, and s3a://) on EMR Serverless. The assumed role configuration applies uniformly, so no additional configuration is required.

On EMR releases before 7.10.0, s3:// and s3n:// use EMRFS and only s3a:// uses S3A. A job that writes with s3:// or s3n:// bypasses the S3A assumed role. On these releases, add the following configuration to force all Amazon S3 schemes through S3A:

spark.hadoop.fs.s3.impl = org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3n.impl = org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.impl = org.apache.hadoop.fs.s3a.S3AFileSystem

Related information

Setting up EMR Serverless (IAM)

Job runtime roles for EMR Serverless

Troubleshooting errors in EMR Serverless