Permission issues while creating s3 batch job

I hope you’d have already followed AWS S3 batch operations permissions setup guide from here:

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-iam-role-policies.html
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html
• https://aws.amazon.com/blogs/aws/new-amazon-s3-batch-operations/

In this process, to get S3 batch operations working, you need to:

1. Setup appropriate permissions as listed in above document links for manifest bucket, source bucket and target bucket
2. Create manifest file aka inventory file, which would contain list of objects for whole bucket/specific prefix
3. Create S3 batch job using the manifest file create in above step

Here is an example of permissions setup required for s3 batch job:

Some of the variables, that need to be replaced in following example:

• inventory_bucket_name: Bucket where manifest file would be stored
• source_account_number: Account which would initiate S3 batch copy
• source_bucket_name: Source bucket from where s3 objects would be copied to target bucket
• target_bucket_name: Target bucket where s3 objects would be copied from source account
• optional_prefix: Any specific prefix(if not whole bucket), from where objects to copied through s3 batch job
• source_ac_s3_batch_copy_role: Role that would be used while creating s3 batch job
• source_acnt_user_role: Role that would be assumed by person, who would create s3 batch job

Here is a quick description of SIDs used in following role policy/bucket policy documents to explain purpose of each of those blocks:

• S3BatchCopyInventory: Manifest bucket to allow write here for storing manifest file
• S3BatchTarget: Source account batch copy role to have write permissions to the whole target bucket or specific prefix
• S3BatchSource: Source account batch copy role to have read permissions on the source bucket or specific prefix
• S3BatchManifestReport: Source account batch copy role to have read and write permissions on the manifest bucket
• DenyAllUnlessApproved: Completely optional but if required to secure target bucket and restrict access only to source account batch copy role and user role which would be by individual creating batch job
• AllowBatchCopyRole: Source account batch copy role to have write permissions on the target bucket
• AllowConsoleBatchJobCreation: This role would allow individual at source account side to create batch job on console, otherwise it’d give permission error

SOURCE Side:

1. Manifest bucket policy to be updated:

    {
        "Sid": "S3BatchCopyInventory",
        "Effect": "Allow",
        "Principal": {
            "Service": "s3.amazonaws.com"
        },
        "Action": "s3:PutObject",
        "Resource": "arn:aws:s3:::<inventory_bucket_name>/*",
        "Condition": {
            "StringEquals": {
                "aws:SourceAccount": "<source_account_number>",
                "s3:x-amz-acl": "bucket-owner-full-control"
            },
            "ArnLike": {
                "aws:SourceArn": "arn:aws:s3:::<source_bucket_name>"
            }
        }
    }

2. Batch Operations Role to be created at source account:

2.1. Permissions:

    {   
        "Version": "2012-10-17”,`
        "Statement": [
            {
                "Action": [
                    "s3:PutObject",
                    "s3:PutObjectAcl",
                    "s3:PutObjectVersionAcl",
                    "s3:PutObjectVersionTagging",
                    "s3:PutObjectTagging"
                ],
                "Resource": [
                    "arn:aws:s3:::<target_bucket_name>/*"
                ],
                "Effect": "Allow",
                "Sid": "S3BatchTarget"
            },
            {
                "Action": [
                    "s3:GetObject",
                    "s3:GetObjectAcl",
                    "s3:GetObjectVersionAcl",
                    "s3:GetObjectVersion",
                    "s3:GetObjectVersionTagging",
                    "s3:GetObjectTagging",
                    "s3:GetBucketLocation"
                ],
                "Resource": [
                    "arn:aws:s3:::source_bucket_name",
                    "arn:aws:s3:::source_bucket_name/<optional_prefix>/*"
                 ],
                "Effect": "Allow",
                "Sid": "S3BatchSource"
            },
            {
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:GetObjectVersion",
                    "s3:GetBucketLocation"
                ],
                "Resource": [
                    "arn:aws:s3:::<inventory_bucket_name>",
                    "arn:aws:s3:::<inventory_bucket_name>/*"
                ],
                "Effect": "Allow",
                "Sid": "S3BatchManifestReport"
            }
        ]
    }

2.2. Batch Operations Role Trust Policy:

    {
       "Version":"2012-10-17",
       "Statement":[
          {
             "Effect":"Allow",
             "Principal":{
                "Service":"batchoperations.s3.amazonaws.com"
             },
             "Action":"sts:AssumeRole"
          } 
       ]
    }

TARGET Side:

3. Target bucket policy to be updated:

    {
        "Version": "2012-10-17",
        "Id": "S3-Batch-Copy-Policy",
        "Statement": [
            {
                "Sid": "DenyAllUnlessApproved",
                "Effect": "Deny",
                "Principal": "*",
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<target_bucket_name>",
                    "arn:aws:s3:::<target_bucket_name>/<optional_prefix>/*"
                ],
                "Condition": {
                    "StringNotLike": {
                        "aws:PrincipalArn": [
                            "arn:aws:iam::<source_account_number>:role/<source_ac_s3_batch_copy_role>",
                            "arn:aws:iam::<source_account_number>:role/<source_acnt_user_role>"
                        ]
                    }
                }
            },
            {
                "Sid": "AllowBatchCopyRole",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::<source_account_number>:role/<source_ac_s3_batch_copy_role>"
                },
                "Action": [
                    "s3:PutObject",
                    "s3:PutObjectAcl",
                    "s3:PutObjectTagging"
                ],
                "Resource": "<target_bucket_name>/<optional_prefix>/*"
            },
            {
                "Sid": "AllowConsoleBatchJobCreation",
                "Effect": "Allow",
                "Principal": {
                    "AWS": [
                        arn:aws:iam::<source_account_number>:role/<source_acnt_user_role>
                    ]
                },
                "Action": [
                    "s3:Get*",
                    "s3:List*"
                ],
                "Resource": [
                    "arn:aws:s3:::<target_bucket_name>",
                    "arn:aws:s3:::<target_bucket_name>/<optional_prefix>/*"
                ]
            }
        ]
    }

Note: These permission are for example and based on the requirement, where source and target accounts are different, setup would would work fine. However permissions can be further adjusted if required.