Why does my Amazon S3 Batch Replication job fail?

4 minute read
0

I want to find out why my Amazon Simple Storage Service (Amazon S3) Batch Replication job fails.

Short description

If your S3 Batch Replication job fails, then the AWS Identity and Access Management (IAM) roles that you use for the job might be incorrectly configured. Complete the following tasks to troubleshoot a S3 Batch Replication job:

  • (Prerequisite) Choose whether to Copy or to Replicate
  • Configure the IAM roles
  • Troubleshoot other causes for job failure

Resolution

Prerequisite

When you use Batch Operations, you can choose to either copy or replicate the objects in a bucket. Before you use Batch Operations, review the following information:

  • For Batch Operations, copy has a limit of 5GB. It also won't allow you to copy a single object that's 5GB or larger. Replication doesn't have this limitation.
  • Replication retains the metadata for the replicated objects, such as the creation date and version ID. The replicated objects also retain object level access control lists (ACLs). Replication allows you to change the storage class of the replicated objects. Copy doesn't allow you to retain this information or change the storage class.
  • To use Copy, the source and destination bucket must be in the same AWS Region.

For more information, see S3 Batch Replication considerations.

Configure the IAM roles

The Amazon S3 Replication rule on the source bucket is the basis for the S3 Batch Replication job. To create a S3 Batch Replication job, define the objects to replicate and (optionally) configure the reports. The S3 Batch Replication job takes all other settings from the Replication rule that's configured on the source S3 bucket. If you don't configure a replication rule on the source bucket, then your Batch job for replication fails.

Incorrect configuration of the IAM roles causes a S3 Batch Replication job to fail. A S3 Batch Replication job uses the following IAM roles:

After you configure the IAM roles, configure the replication rule for the bucket that's appropriate for your replication scenario. Then, configure the S3 Batch Replication job.

Troubleshoot other causes for job failure

Issues with the Batch role

  • The Batch role must have PUT permissions for the manifest or configuration. It also must have GET permissions for the manifest from the bucket that stores the manifest. If you use the Save Batch Operations manifest option, then you must have GET and PUT permissions for the manifest file. Configure the bucket policy of the manifest to not deny any actions that the Batch role requires. For more information, see Configuring IAM policies for Batch Replication.
  • If you use a cross account bucket, then the bucket policy must have permissions for the Batch role. These permissions are in addition to the IAM role policy.
  • If you use AWS Key Management Service (AWS KMS) or object level encryption for the bucket, then the AWS KMS key must include the Batch role. The AWS KMS key must allow the batch role to download and upload data from the bucket.
  • If you use a user supplied manifest, then your manifest must have the correct permissions on the IAM role. If it's a cross account bucket, then bucket policy must include the permissions. If the bucket is AWS KMS encrypted, then the AWS KMS key policy must give the Batch role permission to download the manifest.

Issues with the Replication role

If a Batch Job fails with an error rate over 50%, then it's likely due to an issue in the Replication role configuration. If you use the same role for Batch and Replication jobs, then allow Amazon S3 and Amazon S3 Batch Operations services to assume the role. Make sure that the configuration allows both permission sets.

Note: It takes time to prepare a manifest for Amazon S3. If your Batch job is in the "Preparing" stage or at the same percentage of completion for several hours, then contact AWS Support.

Related information

How do I troubleshoot Amazon S3 Batch Operations issues?

AWS OFFICIAL
AWS OFFICIALUpdated 10 months ago
No comments