AWS S3 Cross Replication - FAILED replication status for prefix

Question

Hi there,  
  
We are utilizing cross-region replication to replicate a large bucket with tens of millions of objects in it to another AWS account for backup purposes.   
  
Originally, we had configured the replication rules to replicate the entire bucket. However, we recently noticed that some objects were not being replicated to the destination bucket in the backup account and appeared with a replication status of FAILED (screenshot: https://pasteboard.co/IgdSZf3.png).   
  
Thinking that perhaps the bucket had too many objects in it and that perhaps CRR was not capable of reliably replicating an entire bucket with that many objects, we created multiple replication rules at the prefix level (i.e. instead of one CRR rule for "bucket-name" we created ~10 for each "subfolder" prefix in the bucket, eg "bucket-name/subfolder1" "bucket-name/subfolder2" "bucket-name/subfolder3"). After doing so, and doing some testing, I have noticed that replication is working fine in all but one of the bucket prefixes. Objects under this prefix can't be replicated, and the replication status shows as FAILED for each new object added to the bucket. This particular prefix has a lot of objects under it.  
  
Replication is working for this bucket for certain prefixes, so it's obviously not a policy or permissions issue. What else can we do to troubleshoot this so we can get CRR working reliably for this bucket?

Answer

Thanks, we're not on a paid support plan at the moment, but fortunately, I don't think we'll need to create a ticket for this. I believe I have FINALLY figured this out.  
  
This issue appears to have been caused by public access settings on the destination bucket. The objects in the problematic "subfolder" in the source bucket are public (which is intentional), but are not supposed to be public in the destination bucket. When I disabled the "Block public access to buckets and objects granted through new access control lists (ACLs)" setting, replication started working (screenshot: https://pasteboard.co/Ii4SQJD.png). Reference from AWS docs on what I'm talking about: https://docs.aws.amazon.com/AmazonS3/latest/user-guide/block-public-access-bucket.html. I believe these settings were somewhat recently introduced.  
  
I've confirmed that the ACL on the replicated object does not, in fact, grant public read access to the object in the destination bucket once it has been replicated. There must be some process that takes place behind the scenes that copies over the original ACL for the object (which the public access policy must block, since it sees it as a "new" ACL that grants public read permission) and then changes the ownership of the object to the destination bucket, probably by changing the ACL.

Answer

Hi,  
  
Do you have a support plan with AWS? You would need to submit a support ticket and provide couple of keys that failed replication so that we could check and see exactly what happened.   
  
Regards,  
Yuriy  
AWS S3

Answer

Marked as "answered" - see above.  
  
Edited by: gbdan on Jun 5, 2019 5:16 PM

AWS S3 Cross Replication - FAILED replication status for prefix

Relevant content