Why aren't my Amazon S3 objects replicating when I set up replication between my buckets?

8 minute read

I set up cross-Region replication (CRR) or same-Region replication (SRR) between my Amazon Simple Storage Service (Amazon S3) buckets. However, objects aren't replicating to the destination bucket.

Resolution

To troubleshoot S3 objects that aren't replicating to the destination bucket, check the different types of permissions for your bucket. Also, check the public access settings and bucket ownership settings.

Tip:

Upload an object to the source bucket to test the replication after each configuration change. It's a best practice to change one configuration at a time to identify any replication setup issues.
Activate the s3:Replication:OperationFailedReplication event type notification for replication failures. The error code might help you determine the cause of the failure.

After you resolve the issues that caused the replication to fail, there might be objects in the source bucket that weren't replicated. By default, S3 replication doesn't replicate existing objects or objects with a replication status of FAILED or REPLICA. To check the objects' replication statuses, see How do I view objects that failed replication from one Amazon S3 bucket to another? Use S3 Batch Replication to replicate the objects.

Grant minimum Amazon S3 permissions

Confirm that the AWS Identity Access Management (IAM) role that you used in the replication rule has the correct permissions. If the source and destination buckets are in different AWS accounts, then confirm that the destination account's bucket policy grants sufficient permissions to the replication role.

The following example IAM policy has the minimum permissions that are required for replication:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::SourceBucket"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::SourceBucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateTags"
            ],
            "Resource": "arn:aws:s3:::DestinationBucket/*"
        }
    ]
}

Note: Replace SourceBucket and DestinationBucket with the names of your S3 buckets.

Based on the replication rule options, you might need to grant additional permissions.

The IAM role must have a trust policy that allows Amazon S3 to assume the role to replicate objects:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Grant additional Amazon S3 permissions

If you set the replication rule to Change object ownership to the destination bucket owner, then the IAM role must have the s3:ObjectOwnerOverrideToBucketOwner permissions. Place the permission on the S3 object resource:

{
    "Effect": "Allow",
    "Action": [
        "s3:ObjectOwnerOverrideToBucketOwner"
    ],
    "Resource": "arn:aws:s3:::DestinationBucket/*"
}

The destination account must also grant the s3:ObjectOwnerOverrideToBucketOwner permission in the bucket policy:

{
    "Sid": "1",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"
    },
    "Action": [
        "s3:ObjectOwnerOverrideToBucketOwner"
    ],
    "Resource": "arn:aws:s3:::DestinationBucket/*"
}

Note: If the destination bucket's object ownership is Bucket owner enforced, then you don't need Change object ownership to the destination bucket owner in the replication rule. The change occurs by default.

If you activated delete marker replication on the replication rule, then the IAM role must have the s3:ReplicateDelete permissions:

{
    "Effect": "Allow",
    "Action": [
        "s3:ReplicateDelete"
    ],
    "Resource": "arn:aws:s3:::DestinationBucket/*"
}

If the destination bucket is in another account, then the destination bucket owner must also grant the permission in the bucket policy:

{
    "Version": "2012-10-17",
    "Id": "PolicyForDestinationBucket",
    "Statement": [
        {
            "Sid": "Stmt1644945277847",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"
            },
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateTags",
                "s3:ObjectOwnerOverrideToBucketOwner",
                "s3:ReplicateDelete"
            ],
            "Resource": "arn:aws:s3:::DestinationBucket/*"
        }
    ]
}

Note: Replace arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role with the ARN of your replication role. Replace DestinationBucket with the name of your S3 bucket.

Grant AWS KMS permissions

If you encrypted the bucket's source objects with an AWS Key Management Service (AWS KMS) key, then the replication rule must include AWS KMS encrypted objects.

Complete the following steps:

Open the Amazon S3 console.
Choose the S3 bucket that contains the source objects.
On the Management tab, select a replication rule.
Choose Edit.
Under Encryption, choose Replicate objects encrypted with AWS KMS.
Under AWS KMS key for encrypting destination objects, select an AWS KMS key. The default option is to use the AWS KMS key (aws/S3).

For more information, see Example policies - Using SSE-S3 and SSE-KMS with replication.

Important: If the destination bucket is in a different account, then specify an AWS KMS customer managed key that's owned by the destination account. The default aws/S3 key encrypts the objects with the AWS managed key that the source account owns. Because you can't share the AWS managed key with another account, the destination account can't access the objects in the destination bucket.

Grant additional AWS KMS permissions for cross-account scenarios

To use the destination account's AWS KMS key to encrypt the destination objects, the destination account must grant the replication role in the key policy:

{
    "Sid": "AllowS3ReplicationSourceRoleToUseTheKey",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"
    },
    "Action": [
        "kms:GenerateDataKey",
        "kms:Encrypt"
    ],
    "Resource": "*"
}

Note: If you use an asterisk (*) for Resource in the AWS KMS key policy, then the policy grants permission only to the replication role. The policy doesn't allow the replication role to expand its permissions.

Also, you must add the following minimum permissions to the replication role's IAM policy from the source account:

[  
    {  
        "Effect": "Allow",  
        "Action": [  
            "kms:Decrypt",  
            "kms:GenerateDataKey"  
        ],  
        "Resource": [  
            "SourceKmsKeyArn"  
        ]  
    },  
    {  
        "Effect": "Allow",  
        "Action": [  
            "kms:GenerateDataKey",  
            "kms:Encrypt"  
        ],  
        "Resource": [  
            "DestinationKmsKeyArn"  
        ]  
    }  
]

By default, the AWS KMS key policy grants the root user full permissions to the key. You can delegate these permissions to other users in the same account. You can also use an IAM policy to grant the replication role permissions to the source KMS key. This is sufficient unless there are deny statements in the source KMS key policy.

Check for explicit deny and conditional allow statements

If your objects still aren't replicating after you validated the permissions, then check for explicit deny statements that can cause replication to fail.

Remove deny statements in the destination bucket policy or AWS KMS key policy that restrict access to the following:

Specific CIDR ranges,
Virtual private cloud (VPC) endpoints
S3 access points

Remove deny statements or permissions boundaries that are attached to the IAM role. Also, remove deny statements in AWS Organizations service control policies (SCPs) that are attached to either the source or destination accounts.

Tip: Before you remove any explicit deny statements, confirm the reason for the deny. Also, determine whether the statement affects data security.

Check for Amazon S3 bucket keys

If the source or destination KMS keys grant permissions based on the encryption context, then check whether S3 Bucket Keys are used. If the buckets use Bucket Keys, then the encryption context must be for the bucket level resource:

"kms:EncryptionContext:aws:s3:arn": [     "arn:aws:s3:::SOURCE_BUCKET_NAME"
     ]
"kms:EncryptionContext:aws:s3:arn": [
     "arn:aws:s3:::DESTINATION_BUCKET_NAME"
     ]

Note: Replace SOURCE_BUCKET_NAME and DESTINATION_BUCKET_NAME with the names of your source and destination buckets.

If Bucket Keys aren't used for the source or destination buckets, then the encryption context must be the object level resource:

"kms:EncryptionContext:aws:s3:arn": [     "arn:aws:s3:::SOURCE_BUCKET_NAME/*"
     ]
"kms:EncryptionContext:aws:s3:arn": [
     "arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
     ]

Note: Replace SOURCE_BUCKET_NAME and DESTINATION_BUCKET_NAME with the names of your source and destination buckets.

Check for object ACLs and Block Public Access

Check whether the source and destination buckets use access control lists (ACLs). If the object includes an ACL that allows public access but the destination bucket uses Block Public Access, then replication fails.

Confirm source object ownership

If another account uploaded the objects in the source bucket, then the source account might not have permission to those objects. Check the source bucket to determine whether ACLs are deactivated.

If the ACLs are deactivated on source bucket, then the source account is the owner of all objects in the bucket. If ACLs aren't deactivated on the source bucket, then check whether object ownership is set to Object owner preferred or Bucket owner preferred. If ownership is set to Bucket owner preferred, then the source bucket objects must have the bucket-owner-full-control ACL permission.

The source account can deactivate ACLs to take ownership of all objects in their bucket. Most use cases don't require that you use ACLs to manage access. It's a best practice to use IAM and bucket policies to manage access to S3 resources. To deactivate ACLs on your S3 bucket, see Controlling ownership of objects and disabling ACLs for your bucket. Your bucket and IAM policies must grant sufficient permissions so that you don't affect Amazon S3 access when you deactivate ACLs.

Specify the correct replication rule filter

Be sure that you correctly specified the replication rule filter.

If you specify a rule filter with a combination of a key prefix and object tags, S3 performs a logical AND operation to combine the filters. The rule applies to a subset of objects with a specific key prefix and specific tags.