S3 Replication FAILED on some objects

1

I have setup Cross Region and Cross Account S3 replication. Both source and destination buckets are setup with the same CloudFormation template (same permissions and class), the only difference being the destination doesn't have a replication rule. Here is the replication rule on the source.

{
  "Role": "arn:aws:iam::{SourceAccount}:role/{SourceBucket}",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "DeleteMarkerReplication": { "Status": "Disabled" },
      "Filter" : {},
      "Destination": {
          "Bucket": "arn:aws:s3:::{DesitnationBucket}",
          "Account":"{DesitnationAccount}",
          "AccessControlTranslation":{
            "Owner":"Destination"
        },
          "Metrics": {
            "Status": "Enabled"
        }
      }
    }
  ]
}

The replication works. However, out of the about 1 000 000 objects I have, 66 000 show the status as failed to replicate. When I run an S3 batch job on the bucket to replicate, it fails as well, and from the output, it states the error as "SrcObjectNotEligible" "Object is not eligible for replication. Object or object tags do not match replication configuration."

As you can see in the above rule, I have no filters applied, and all objects should replicate. I do have versioning and object lock enabled on the bucket, and the objects are not encrypted (this is not preferred. However, the source that uploads to the bucket requires this to be off.)

As per https://aws.amazon.com/premiumsupport/knowledge-center/s3-list-objects-failed-replication/#:~:text=Amazon%20S3%20inventory%20report,-Amazon%20S3%20inventory&text=To%20find%20objects%20that%20failed,source%20bucket%2C%20which%20triggers%20replication

The only way to resolve a failure is "To find objects that failed replication, filter a recent report for objects with the replication status of FAILED. Then, you can initiate a manual copy of the objects to the destination bucket. Or, you can re-upload the objects to the source bucket, which triggers replication."

As I would like the status to be "Replicated" I am not sure a manual copy will work. I downloaded the file from S3 and proceeded to overwrite the existing one. It does start the process of replicating it again but also fails.

Does anyone have other options for me?

  • Did you ever resolve this issue? I'm seeing the same thing, and I can find no configuration difference between objects that succeed and those that don't. Re-running the same batch operation succeeds in same cases, but in others continues to see a failure rate.

Bobk
asked a year ago1379 views
1 Answer
-1

Hi THere

Please take a look at the list of reasons why an object wouldn't get replicated here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-what-is-isnot-replicated.html#replication-what-is-not-replicated

profile pictureAWS
EXPERT
Matt-B
answered a year ago
  • I have checked those out already, and none seem to apply. There are also no lifecycle policies applied to either source or destination buckets. The source and destination have also been the same since the first setup.

  • Can you do a "aws s3api head-object --bucket <bucket-name> --key <full key prefix and name>" command on one of the failed objects and post the output? If you need help with that CLI command see https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/head-object.html

  • Here is the output from 1

    AcceptRanges: bytes
    ContentLength: 1978
    ContentType: application/octet-stream
    ETag: '"d3a7e36d53e05328a4560893fdc7e00b"'
    LastModified: '2022-09-29T20:17:10+00:00'
    Metadata:
      arch-block-size: '1894'
    ObjectLockLegalHoldStatus: OFF
    ObjectLockMode: COMPLIANCE
    ObjectLockRetainUntilDate: '2023-01-20T20:06:48+00:00'
    ReplicationStatus: FAILED
    VersionId: AHYyrnOSStfFBsOfT8WMOatsVEGX0M97
    
  • Matt-B, any update here? I'm seeing the same issue.

  • Hi There

    I see tat ObjectLockMode is set to COMPLIANCE. Please ensure that the destination bucket also has ObjectLock mode set to the same. If the source bucket has S3 Object Lock enabled, the destination buckets must also have S3 Object Lock enabled or replication to work

    See https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html#replication-requirements

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions