Why does my Amazon S3 Batch Replication job fail?

Short description

Batch replication can fail because of issues with permissions, policies, or a job configuration.

To prevent a large number of unsuccessful operations, Amazon S3 sets a task-failure threshold on every Batch Operations job. Amazon S3 monitors the task failure rate after the job runs at least 1,000 tasks. If a job exceeds the 50% failure rate, then the job fails. Use the following troubleshooting to determine and resolve the issue. Then, resubmit the job.

Resolution

Batch role permissions

Grant the following permissions to the batch role:

• Provide PUT permissions for the manifest or configuration.
• Provide GET permissions for the manifest with the bucket that stores the manifest.
• If you use your own manifest, then grant the batch role permissions to access the manifest.
• Grant the s3:CreateJob permission to create the batch job.
• If you apply a job tag option, then also grant the s3:PutJobTagging permission.

Note: To save batch operations, the batch role must have GET and PUT permissions for the manifest file. For more information, see Configuring an AWS Identity and Access Management (IAM) role for S3 Batch Replication.

Replication role permissions

When a batch job exceeds the 50% failure rate, then there might be an issue in the replication role configuration.

When you're configuring your replication role, use the following settings:

• If you use the same role for batch and replication jobs, then grant both Amazon S3 and Amazon S3 Batch Operations permission to assume the role. 
• Include the minimum required permissions for replication. For more information, see Why aren't my Amazon S3 objects replicating when I set up replication between my buckets?
• Grant additional permissions for AWS Key Management Service (AWS KMS), bucket keys, object ownership change, and delete marker replication.
• Block public access settings.
• Review the permission boundaries that might be attached to the IAM roles because the boundaries might cause replication failure.

Source and destination bucket policy

Use the following bucket policy settings:

• If the destination bucket is in another AWS account, then grant permission in the bucket's source replication rule for the replication role to replicate objects. These permissions are in addition to the IAM role policy. Also, don't configure the destination bucket as Requester Pays.
• By default, the source bucket owner doesn't have permission to access the objects that other accounts create. The replication configuration replicates only the objects that the source bucket owner can access. To resolve this issue, either deactivate the access control lists (ACLs) in object ownership controls, or change the ownership of the objects to the source account.
• If you store the batch job's manifest in a cross-account bucket, then allow the bucket account to access the manifest. Add the permission to the bucket policy for the batch role.

AWS KMS permission for encrypted objects

Use the following AWS KMS settings for encrypted objects:

• If you use AWS KMS or object-level encryption for the manifest and report bucket, then include the batch role in the key. The AWS KMS key must allow the batch role to download and upload data from the bucket.
• If the replication includes an encrypted source or destination bucket, then allow access to the replication and batch roles with the AWS KMS keys. Grant the batch role permission to download the manifest. 
• If you grant permission based on encryption context, then include the ARN that's based on the bucket key configuration. See Why aren't my Amazon S3 objects replicating when I set up replication between my buckets?
• Use Amazon S3 Inventory reports that are AWS KMS-encrypted. Amazon S3 doesn't support manually generated AWS KMS encrypted manifest files for batch operations.

Explicit Deny statements

If the bucket policy's Deny statement restricts actions based on the network, then Amazon S3 can't perform the replication. Your Amazon S3 replication doesn't use the virtual private cloud (VPC) endpoint or Amazon S3 access points that you configured for private access.

To remove restrictions, take the following actions:

• Configure the manifest's bucket policy not to deny any actions that the batch role requires.
• If you use private access when you create the S3 batch job, then grant the VPC endpoint permission to create the batch job.
• Verify that you didn't deny the required replication permissions in IAM, AWS Organizations service control policies (SCPs), bucket policies, or AWS KMS policies.

Batch replication job configuration errors

Configuration issues can cause the following error responses.

Manifest generation found no keys matching the filter criteria

Don't include objects in the manifest that are in Amazon S3 Glacier or Amazon S3 Glacier Deep Archive because a replication operation doesn't find them.

If the replication rule's prefix filter or filter criteria in the batch replication job doesn't match a prefix in the bucket, then restore the objects. After you restore the objects, perform the replication again.

SrcGetObjectNotPermitted or SrcHeadObjectNotPermitted

You get the SrcGetObjectNotPermitted or SrcHeadObjectNotPermitted error when the IAM role that you configured in the replication rule doesn't have permission to access objects. You can also get one of the errors when the IAM role can't retrieve object metadata from the source bucket. To resolve these issues, add the s3:GetObjectVersionForReplication permission in the IAM role policy.

For more information, see Setting up permissions to create replication rules.

DstPutObjectNotPermitted

When you get the DstObjectNotPermitted error, Amazon S3 can't replicate objects to the destination bucket. To resolve this issue, grant the s3:ReplicateObject or s3:ObjectOwnerOverrideToBucketOwner permission in both the IAM role policy and the destination S3 bucket policy.

Batch job ran successfully but the number of objects expected in destination bucket is not the same

This error occurs when there's a mismatch between objects that are listed in the manifest and the filter. To resolve this issue, fix the typos in the filter that you selected when you created the job or in the replication rule configuration.

400 InvalidRequest: Task failed due to missing VersionId

This error occurs when the job encounters an object in the manifest with an empty version ID field. To resolve this issue, convert the empty version IDs to null strings. 

Reasons for failure. The job report could not be written to your report bucket. Please check your permissions.

Amazon S3 doesn't support object locks on the bucket that you designated to store the batch completion report. Instead, store the report in an S3 bucket without object lock.

Objects skipped without an error message

Amazon S3 doesn't replicate an object in a source bucket that's a replica from another replication configuration.

Also, Amazon S3 doesn't replicate objects to the same destination bucket where it's successfully completed a replication. Instead, Amazon S3 skips the object without an error message.

Monitoring

After a batch replication job completes, Amazon S3 generates a batch completion report in the destination bucket or prefix that you specified. This report provides detailed information that includes failures and associated error messages. Review the batch complete report to identify objects or operations that failed during the replication process and their associated error or failure reasons.

Use Amazon S3 Event Notifications to notify you when objects don't replicate to their destination AWS Region.

Note: It takes time to prepare a manifest for Amazon S3. However, if your batch job is in the Preparing stage or at the same percentage of completion for several hours, then contact AWS Support.

Related information

How do I troubleshoot Amazon S3 Batch Operations issues?