S3 Replication FAILED on some objects

I have setup Cross Region and Cross Account S3 replication. Both source and destination buckets are setup with the same CloudFormation template (same permissions and class), the only difference being the destination doesn't have a replication rule. Here is the replication rule on the source.

{
  "Role": "arn:aws:iam::{SourceAccount}:role/{SourceBucket}",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "DeleteMarkerReplication": { "Status": "Disabled" },
      "Filter" : {},
      "Destination": {
          "Bucket": "arn:aws:s3:::{DesitnationBucket}",
          "Account":"{DesitnationAccount}",
          "AccessControlTranslation":{
            "Owner":"Destination"
        },
          "Metrics": {
            "Status": "Enabled"
        }
      }
    }
  ]
}

The replication works. However, out of the about 1 000 000 objects I have, 66 000 show the status as failed to replicate. When I run an S3 batch job on the bucket to replicate, it fails as well, and from the output, it states the error as "SrcObjectNotEligible" "Object is not eligible for replication. Object or object tags do not match replication configuration."

As you can see in the above rule, I have no filters applied, and all objects should replicate. I do have versioning and object lock enabled on the bucket, and the objects are not encrypted (this is not preferred. However, the source that uploads to the bucket requires this to be off.)

As per https://aws.amazon.com/premiumsupport/knowledge-center/s3-list-objects-failed-replication/#:~:text=Amazon%20S3%20inventory%20report,-Amazon%20S3%20inventory&text=To%20find%20objects%20that%20failed,source%20bucket%2C%20which%20triggers%20replication

The only way to resolve a failure is "To find objects that failed replication, filter a recent report for objects with the replication status of FAILED. Then, you can initiate a manual copy of the objects to the destination bucket. Or, you can re-upload the objects to the source bucket, which triggers replication."

As I would like the status to be "Replicated" I am not sure a manual copy will work. I downloaded the file from S3 and proceeded to overwrite the existing one. It does start the process of replicating it again but also fails.

Does anyone have other options for me?