How do I set up cross-account access for EMR File System?

Short description

To set up cross-account access for EMRFS, use one of the following methods:

• Add an Amazon Simple Storage Service (Amazon S3) bucket policy
• Use Amazon S3 Access Grants
• Assume an AWS Identity and Access Management (IAM) role in the destination account.

Resolution

Add an Amazon S3 bucket policy

Add an Amazon S3 bucket policy for the destination account that grants access to the EMRFS account. By default, the bucket account owns the objects that you create. To control ownership of the bucket objects, see Controlling ownership of objects and turning off access control lists (ACLs) for your bucket.

Use Amazon S3 Access Grants

For Spark access to a bucket that another account owns, you can use Amazon S3 Access Grants for Amazon EMR version 6.15.0 or later. To manage access, configure the access grant in the account that owns the bucket.

For more information about how to use Amazon S3 Access Grants to manage access, see the following documentation:

• Launch an Amazon EMR cluster with Amazon S3 Access Grants for Amazon Elastic Compute Cloud (Amazon EC2)
• Launch an Amazon EMR Serverless application with Amazon S3 Access Grants for data management
• Launch an Amazon EMR on Amazon Elastic Kubernetes Service (Amazon EKS) cluster with Amazon S3 Access Grants for data management

Assume an IAM role in the destination account

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

For Amazon EMR on Amazon EC2, use a security configuration to assume the IAM role.

For Amazon EMR on Amazon EKS, you must specify the IAM role in the Amazon EMR on Amazon EKS job configuration properties.

For EMR Serverless, you must specify the IAM role in the EMR Serverless job configuration properties. For Amazon EMR Serverless 6.11.0 or later, use the EMR Serverless BucketLevelAssumeRoleCredentialProvider to specify multiple roles to access different buckets.

To use a security configuration to assume an IAM role in the destination account for Amazon EMR on Amazon EC2, complete the following steps:

1. Create an IAM role in the destination account that owns the EMRFS data in the Amazon S3 bucket. The Amazon EMR application assumes this role in the source account.

2. Add a trust policy that grants permissions for the source cluster's instance profile role to assume the destination account IAM role:

   {  "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::<example-source-account-id>:role/EMR_EC2_DefaultRole"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

   Note: Replace example-source-account-id with your source account ID. For more information, see Configure roles.

3. Use the AWS CLI to create a security configuration with role mapping for the destination account IAM role. Then, provide the following JSON object:

   {  "AuthorizationConfiguration": {
       "EmrFsConfiguration": {
         "RoleMappings": [
           {
             "Role": "arn:aws:iam::<example-destination-account-id>:role/example-role",
             "IdentifierType": "Prefix",
             "Identifiers": [
               "s3://example-bucket/"
             ]
           }
         ]
       }
     }
   }

   Note: Replace example-destination-account-id with your destination account ID, example-role with your destination account role, and example-bucket with the destination account Amazon S3 bucket.

4. Create an IAM policy in the Amazon EMR source account. The IAM policy must grant permissions for AWS Security Token Service (AWS STS) to assume the destination account IAM role.
   Note: For Amazon EMR on Amazon EKS and EMR Serverless, attach the following IAM policy to the job runtime role in the source account. Make sure the job runtime role is in the source account that runs the Amazon EMR application.
   Example policy:

   {  "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Resource": "arn:aws:iam::<example-destination-account-id>:role/example-role"
       }
     ]
   }

5. Attach the IAM policy to the Amazon EMR Amazon EC2 instance profile, for example EMR_EC2_DefaultRole.

6. Launch an Amazon EMR cluster and specify the security configuration. If the destination bucket uses server-side encryption (SSE) with AWS Key Management Service (AWS KMS), then the assumed role must be a key user.

7. (Optional) If the previous methods aren't sufficient for your use case, then create a custom credentials provider to assume IAM roles.