By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

AWS Backup Jobs issues with RDS

0

We are trying to move all backup operations to the AWS Backup service using a backup tagging strategy.

We have a tag to enable backups backup_enbled=true and a tag to select a backup plan backup_plan=daily_30 - These two tags together will backup the resource every day and expire the backups after 30 days.

A few issues has come up with this method:

Backup jobs have started failing with this error message.

Backup job <JOB-ID> could not start because it is either inside or too close to the automated backup window configured in RDS instance.

When looking at the RDS instance in the RDS console we see automated snapshots being created successfully so it seems like the RDS service somehow takes precedence over AWS Backup even though the backup plan is configured in AWS Backup.

If we try to manually add a backup plan through the RDS console we see this message, so everything should be running from AWS Backup.

Your automated backups are being managed in a backup plan within AWS Backup. You can still take snapshots manually from this page.

It looks like the automatic snapshots being created from the RDS service starts at 00:00 (UTC+02:00) so we could try to shedule the backups from AWS Backup later. We would however like to keep all backups in AWS Backup to centralize administration.

Are we missing something?

Topics
StorageMigration & ModernizationAnalyticsDatabase
Tags
Backup & RecoveryAmazon Relational Database ServiceAWS Backup
Language
English
rePost-User-7084652
asked 2 years ago4.6K views
3 Answers
0

As described in this document, this may occur when an RDS maintenance period or RDS automatic backup period approaches.
Assuming that the backups themselves are managed correctly by AWS Backup, I thought it necessary to check the maintenance windows, etc.
https://repost.aws/knowledge-center/aws-backup-troubleshoot-scheduled-backup-plans

This can happen when the RDS maintenance window or the RDS automated backup window is approaching. In AWS Backup, RDS backups aren't allowed within an hour before the RDS maintenance window or the RDS automated backup window. Therefore, be sure that your backup plans for RDS databases are scheduled more than an hour apart from the RDS maintenance window and the RDS automated backup window. This time frame is extended to 4 hours for Amazon FSx.

profile picture
EXPERT
Riku_Kobayashi
answered 2 years ago
  • rePost-User-7084652
    2 years ago

    The maintenance window for one of the databases is set to Monday, 00:00 UTC with a duration of 3 hours. I don't think this should cause backups to fail every day. I can't seem to find the Backup Window settings in the RDS console as the backup is manged by AWS Backup as per the message in the Backup section on the Modify page.

    This instance has a backup plan in AWS Backup. To view and manage it, go to the Backup plans page in AWS Backup.

  • Riku_Kobayashi EXPERT
    2 years ago

    Please use the following AWS CLI to check if the backup window is not configured on the RDS side. In rare cases, the contents of the management console and AWS CLI may differ, so it is a good idea to check.
    https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/describe-db-instances.html

    aws rds describe-db-instances --db-instance-identifier mydbinstancecf
  • rePost-User-7084652
    2 years ago

    It sure looks like there is a backup window configured when using above command. As I don't think you can set the backup window to null should we just try to find a time during mid-day when we're sure no backup plans from AWS Backup would run? And can we make sure RDS doesn't suddently schedule a backup during this time?

  • rePost-User-7084652
    2 years ago

    We tried changing the backup window to midday but it seems to have reverted during the weekend. The BackupRetentionPeriod is controlled by AWS Backup and matches our backup plan.

0

"If you manage your backups in AWS Backup, you can't enable automated backups. For more information, see Using AWS Backup to manage automated backups." https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html#USER_WorkingWithAutomatedBackups.BackupWindow

Maybe try and disable the automated backups and see if you can then enable AWS Backup?

profile pictureAWS
Bent_T
answered 2 years ago
  • rePost-User-7084652
    2 years ago

    As far as I can tell, the RDS automated backups are disabled. When I try to set the backup-retention-period to 0 using the CLI I get the following message

    > aws rds modify-db-instance --db-instance-identifier <instance-identifier> --backup-retention-period 0

An error occurred (InvalidParameterValue) when calling the ModifyDBInstance operation: Your RDS instance <instance-identifier> is associated with an AWS Backup resource with id <resource-arn>. You can leave BackupRetentionPeriod blank, or you can specify it only with the current value 30. For more details, see the AWS Backup documentation.

    A value of 30 matches the retention period set by AWS Backup using the resource tagging strategy.

0

Therefore, be sure that your backup plans for RDS databases are scheduled more than an hour apart from the RDS maintenance window and the RDS automated backup window.

The automated backup windows are listed here per region https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html#USER_WorkingWithAutomatedBackups.BackupWindow

profile pictureAWS
Bent_T
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content