RDS Backup & Restore SP Failing with Error - Provided Token Is Expired

0

We had scheduled, daily auto back-up of our SqlServer Db on RDS, to S3 bucket for at least last 6 years. It was working fine since then. Suddenly it stopped working, which means we don't see any Backup in our S3 bucket since 24th March. Up on diagnosing the problem, we realized that it is failing since then and the error is

STEP1 exec msdb.dbo.rds_restore_database @restore_db_name='RestoreDbFromS3', @s3_arn_to_restore_from='arn:aws:s3:::awsbucketName/SqlServerDb.bak';

STEP 2 exec msdb.dbo.rds_task_status @task_id=7;

Response indicates Error with following Task Description [2022-05-28 12:51:22.030] Task execution has started. [2022-05-28 12:51:22.237] Aborted the task because of a task failure or an overlap with your preferred backup window for RDS automated backup. [2022-05-28 12:51:22.240] Task has been aborted [2022-05-28 12:51:22.240] The provided token has expired.

We studied a lot to identify the root cause and solution but could not find anything accurately relevant. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/SQLServer.Procedural.Importing.html#SQLServer.Procedural.Importing.Native.Troubleshooting

above link shows troubleshooting options as per the error responses, but this does not include the error response that we are getting.

Note: between 25th & 26th March, our aws instance was suspended for couple of hours due to delayed payment of monthly invoice. we restored the same quickly. Everything on the same aws account is working fine since then, but we just found out that db backup service has impacted as we see the last successful backup available in S3 bucket is of dated 24th March.

We suspect that some token has expired up on account suspension, but are unable to identify which one and how to restore the same back to normal.

Help, Assistance and Guidance would be much appreciated.

1 Answer
0

Hi

I understand that your automated back-ups on RDS to S3 bucket stopped backing up your data after the 24th of March as your account was suspended for a few hours. It could be that when your account was suspended, your resources were also stopped and as a result, your IAM role might be still holding on to the cached credentials before the account suspension. You may refresh the IAM credentials[1] to release the cached credentials and see if that would solve the issue at hand. Please also see this documentation [2] on troubleshooting sql server backup issues. You may also consider reaching out to AWS Premium Support to have them look into your issue.

Reference:

[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html

[2] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/SQLServer.Procedural.Importing.html

I hope this information would be helpful to you.

Winnie
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions