Do you have a support plan with AWS? You would need to submit a support ticket and provide couple of keys that failed replication so that we could check and see exactly what happened.
Thanks, we're not on a paid support plan at the moment, but fortunately, I don't think we'll need to create a ticket for this. I believe I have FINALLY figured this out.
This issue appears to have been caused by public access settings on the destination bucket. The objects in the problematic "subfolder" in the source bucket are public (which is intentional), but are not supposed to be public in the destination bucket. When I disabled the "Block public access to buckets and objects granted through new access control lists (ACLs)" setting, replication started working (screenshot: https://pasteboard.co/Ii4SQJD.png). Reference from AWS docs on what I'm talking about: https://docs.aws.amazon.com/AmazonS3/latest/user-guide/block-public-access-bucket.html. I believe these settings were somewhat recently introduced.
I've confirmed that the ACL on the replicated object does not, in fact, grant public read access to the object in the destination bucket once it has been replicated. There must be some process that takes place behind the scenes that copies over the original ACL for the object (which the public access policy must block, since it sees it as a "new" ACL that grants public read permission) and then changes the ownership of the object to the destination bucket, probably by changing the ACL.
Marked as "answered" - see above.
Edited by: gbdan on Jun 5, 2019 5:16 PM
Does S3 Cross Account replication happens over Public Internet or Private Network of AWS backboneasked 11 days ago
AWS S3 Cross Replication - FAILED replication status for prefixasked 3 years ago
How to implement S3 replication for existing objects in CDK?asked a month ago
Manually set S3 replication statusAccepted Answerasked 2 years ago
AWS S3 replication rules: Number of distinct destination bucket ARNs cannot exceed 22Accepted Answerasked 8 months ago
S3 Hudi Replication and Failoverasked 6 months ago
AWS S3 Replication With Lifecycle Policy - Cross Accountasked 3 years ago
Does AWS S3 CRR/SRR replication order match the order of object creation?asked 3 years ago
How to remove replication statusasked a year ago
Why do we need S3 bucket versioning enabled in order to do replication?Accepted Answerasked 2 years ago