AWS Backup Copy Job is running for almost two months

Question

TL/DR: an AWS Backup Copy Jobs got stuck in the `RUNNING` state for two months, preventing us to perform other Copy Jobs for the same resource, how to cancel the Job?

We are running daily AWS Backup Copy Jobs copying snapshots from `us-east-1` to `ca-central-1` within the same account.

For the last two months we're getting the following error (Python SDK) when trying to copy backups for one particular Aurora cluster:

```
An error occurred (LimitExceededException) when calling the StartCopyJob operation: Too many copy jobs already running for arn:aws:rds:us-east-1:123456789012:cluster:clustername-suffix and ca-central-1
```

I ran the following command to see if there are any Copy Jobs running and, indeed, there are many:

```
$ aws backup list-copy-jobs --by-resource-arn arn:aws:rds:us-east-1:123456789012:cluster:clustername-suffix --query 'CopyJobs[].[CopyJobId,CreationDate,State]'
[
    [
        "093AEB2C-56C9-9A0D-2E9D-6C396CCE5FA8",
        "2024-11-26T00:00:37.543000+00:00",
        "CREATED"
    ],
    [
        "A52BC826-435F-BFE8-D5AA-1D52DC0F6075",
        "2024-11-25T00:00:38.488000+00:00",
        "CREATED"
    ],
...

[
        "51C1F3CC-61CA-30E4-DF44-914568383865",
        "2024-11-01T00:00:37.697000+00:00",
        "CREATED"
    ],
    [
        "E2FAFB23-2C2A-D4B0-7E0A-C77500696E71",
        "2024-10-31T00:00:38.078000+00:00",
        "RUNNING"
    ]
]
```

There are dozens jobs sitting there in the `CREATED` state and one `RUNING`. 
The source backup that was used to create the Job does not exist anymore.

As there is no cli command or API call to cancel a Copy Job, what are our options?

re:Post Agent · Answer

In the situation you've described with an AWS Backup Copy Job stuck in the RUNNING state for an extended period, unfortunately, there isn't a direct way to cancel a Copy Job that's already in progress. AWS Backup doesn't provide a specific API call or CLI command to cancel an ongoing Copy Job.

However, you have a few potential options to address this issue:

1. Contact AWS Support: Given the unusual duration of this Copy Job (running for two months), it's advisable to reach out to AWS Support. They may be able to investigate the root cause and potentially terminate the stuck job from their end.

2. Wait for job completion or failure: Copy Jobs that have progressed to the RUNNING state cannot be cancelled directly. The job will eventually either complete or fail on its own, though in your case, this approach may not be practical given the extended duration.

3. Review and adjust backup policies: To prevent this issue in the future, you might want to review your backup policies and consider implementing timeout mechanisms or adjusting the frequency of your copy jobs.

4. Check for resource constraints: Ensure that there are no resource constraints in the destination region (ca-central-1) that might be causing the job to hang.

5. Terminate the source resource: As a last resort, if possible and safe to do so, terminating the source Aurora cluster might force the copy job to fail. However, this is a drastic measure and should only be considered if absolutely necessary and after consulting with AWS Support.

For the CREATED state jobs, these are likely queued jobs that haven't started due to the ongoing RUNNING job. Once the RUNNING job completes or fails, these should start processing.

In the meantime, to prevent accumulation of more CREATED jobs, you may want to temporarily pause your daily copy job schedule until this issue is resolved.

Remember, it's crucial to work with AWS Support on this issue, as they can provide tailored advice and potentially resolve the stuck job without risking data loss or integrity.  
**Sources**  
[CopyJob - AWS Backup](https://docs.aws.amazon.com/aws-backup/latest/devguide/API_CopyJob.html)  
[CancelJob - AWS Batch](https://docs.aws.amazon.com/batch/latest/APIReference/API_CancelJob.html)  
[Creating backup copies across AWS accounts - AWS Backup](https://docs.aws.amazon.com/aws-backup/latest/devguide/create-cross-account-backup.html)

Leo K · Answer

Contacting support as the bot suggested is probably the fastest way to sort the issue out. But if you aren't under a paid support plan or would like to try to sort the issue out yourself while waiting for support to respond, given that only one specific cluster is having this issue, one thing you could check is if the cluster is encrypted at rest, and if so, if the key policy of the KMS key grants access to the role under which AWS Backup is running, or if the key policy contains an allow statement deferring the authorisation decision to IAM and the IAM role used by AWS backup contains an allow statement granting access to the key.

Another way to check whether KMS errors might be involved is to check if authorisation errors are logged in CloudTrail in the source or destination regions for the role AWS Backup is using. If you have your own CloudTrail logs set up for analysing with Athena, you can filter by `errorcode is not null and coalesce(useridentity.sessioncontext.sessionissuer.username, '')='YourAwsBackupRoleNameHere'` and apply an appropriate date-based filter just to look at the past day or two, to contain costs.

If you haven't got your own CloudTrail logs set up or no facility for analysing them, you can open the CloudTrail console first in the source region, open the event history view, and in the settings for the log view, expose the "Error code" column, so that it's easy to distinguish between errors and successful requests. Then filter the events by Event source matching "kms.amazonaws.com" or by Event name matching "Decrypt", "GenerateDataKey", "ReEncrypt", or "CreateGrant". Look for events that show an error in the Error code column and which are related to the role used by AWS Backup.

If there are no errors in the source region, repeat the process in the target region.

If you find errors, the details of the CloudTrail event should help to pinpoint the source of the issue.

AWS Backup Copy Job is running for almost two months

Add your answer

Relevant content