Partially replicated through S3 replication

Question

Hi all

I created a cross-region S3 replication between two S3 buckets encrypted with two different KMS CMK (SSE-KMS) in an account. The source bucket is the centralized log archive bucket in the LogArchive account created by LZA. From the source bucket, I see the following 7 folders.

```
AWSLogs/ (Config)
CloudWatchLogs/
connectionlogs/ 
firewall/
guardduty/
logs/
vpc-flow-logs/
```

But from the destination bucket, I only see the 4 folders.

```
AWSLogs/ 
CloudWatchLogs/
firewall/
guardduty/
```

What could go wrong?

Kidd Ip · Answer

Would suggest take a look at this:

* Review and update KMS key policies to include the replication role.
* Ensure replication rules cover all prefixes.
* Check replication metrics and logs for errors.
* Manually re-upload or trigger replication for missed objects if needed.

re:Post Agent · Answer

When you're experiencing partial replication between S3 buckets, there are several potential causes to investigate:

1. **Pre-existing objects**: Objects that existed in your source bucket before you set up the replication rule are not automatically replicated. S3 doesn't replicate objects retroactively. The folders missing in your destination bucket might contain only objects that were created before you configured replication.

2. **KMS key permissions**: Since you're using different KMS CMKs for encryption in both buckets, ensure that your replication role has the necessary permissions to use both keys. The replication role needs `kms:GenerateDataKey` and `kms:Encrypt` permissions for the destination KMS key. Check your KMS key policies to verify these permissions are properly configured.

3. **Replication configuration filters**: Your replication rule might have filters (prefixes or tags) that exclude certain folders. Verify that your replication configuration includes all the folders you want to replicate.

4. **Replication time**: Large objects or a large number of objects may take longer to replicate. Check if the missing folders contain particularly large files or a high volume of objects.

5. **Replication status**: You can check the replication status of individual objects in the source bucket to see if replication is pending, failed, or completed.

6. **IAM permissions**: Ensure that the IAM role used for replication has sufficient permissions to read from all folders in the source bucket and write to the destination bucket.

To resolve this issue, I recommend:
- Verifying your replication configuration settings
- Checking the IAM role permissions
- Ensuring proper KMS key policies are in place
- For pre-existing objects, consider using S3 Batch Replication to replicate them

If the missing folders contain only objects created before setting up replication, you'll need to use S3 Batch Replication or manually copy them to trigger replication.  
**Sources**  
[S3 CRR Replication object versioning issue | AWS re:Post](https://repost.aws/questions/QU2a2auv4ZQq6tGQYwuWn2wg/s3-crr-replication-object-versioning-issue)  
[Troubleshooting replication - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-troubleshoot.html)

Partially replicated through S3 replication

Add your answer

Relevant content