How do I troubleshoot issues with software updates in OpenSearch Service?

Short description

You might experience the following issues during software updates:

• High downtime
• Domain eligibility issues
• Data loss

Note: OpenSearch Service software updates are different from OpenSearch version upgrades.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Troubleshoot high downtime

OpenSearch Service software updates use blue/green deployment to minimize downtime. Updates typically complete within minutes but can take several hours to days for systems with a heavy load. The duration of software updates depends on your domain configuration, number of nodes, shard data, and existing requests to the cluster when the update occurs.

To reduce your downtime, use the following best practices:

• Keep traffic on the domain as low as possible during the update process.
• Schedule software updates during the off-peak window to avoid long update periods.
  Note: You can configure a custom off-peak window to change the start time for software updates.
• Make sure that the cluster is in a healthy and active state when you update the software.
• Use Amazon CloudWatch alarms to make sure that your resource usage isn't too high.

Troubleshoot domain eligibility issues

To perform a service software update, your domain must be eligible. For a list of ineligible states and how to troubleshoot them, see When domains are ineligible for an update.

Troubleshoot data loss

OpenSearch Service takes automated snapshots to back up your data in the event of data loss. If you experience data loss or a red cluster status, then use a snapshot to restore your domain.

To proactively back up your data, you can take manual snapshots of your domain.

Important: After OpenSearch Service applies a software update, you can't perform a rollback. If your service update is stuck, then open an AWS Support case.

Check information about your update

To schedule updates, you can initiate a configuration change that runs the blue/green deployment. When the blue/green deployment runs, OpenSearch Service performs the configurations and runs any pending service software updates.

To check the status of a software update, open the OpenSearch Service console, and then choose Notifications. Each notification gives you information about the software update. If the notification severity is Informational, Low, or Medium, then the update is optional. To access optional updates, you must manually update your domain.

Required updates have a notification severity of High or Critical. OpenSearch Service automatically runs required updates within the need by date, during the domain's off-peak window. This update typically occurs within 14 days from its availability. You can still manually update your domain for required updates to get faster access to new features.

You receive a notification when a service software update is available, required, started, completed, or failed. Two weeks before the scheduled update date, OpenSearch Service sends notification emails to the registered email address on the AWS account.

Note: For automatic updates, OpenSearch Service sends notifications when it starts and when it completes the update. If you manually start an update, then OpenSearch Service sends a notification only when the update completes, not when the update starts.

Cancel an update

Important: You can cancel a software update only when the UpdateStatus is PENDING_UPDATE and before the AutomatedUpdateDate date.

To cancel a scheduled software update, run the following cancel-service-software-update AWS CLI command:

aws opensearch cancel-service-software-update --domain-name mydomainname

Note: Replace mydomainname with your domain name.

Related information

Why is my OpenSearch Service domain upgrade taking so long?