What is a blue/green deployment for an OpenSearch Service cluster?

Short description

When you change your OpenSearch Service cluster configuration, OpenSearch Service might start a blue/green deployment. Blue/green deployment reduces downtime and smooths the transition of version or configuration changes. During a blue/green deployment, the cluster state changes to Modifying as OpenSearch Service creates a new environment.

During the new environment creation process, the number of nodes in your cluster doubles. The total number of nodes is equal to the node count in the previous and new environment. After OpenSearch Service provisions the new nodes, the cluster state returns to Active, and OpenSearch Service migrates data to the new nodes. After data migration completes, OpenSearch Service terminates the previous nodes.

Note: OpenSearch Service quotas don't affect the nodes that OpenSearch Service creates during blue/green deployment. For example, if a cluster has 70 nodes, then it can have 140 nodes during blue/green deployment, even if the quota is 80 nodes.

OpenSearch Service performs a series of validation checks to confirm that your domain is eligible for an update before it starts blue/green deployment. If a check fails, then you must resolve the issue before you retry the configuration change.

Resolution

Note: It's a best practice to schedule blue/green deployments during your domain's off-peak window.

Identify what causes a blue/green deployment

Before you change your configuration, check whether the change can cause a blue/green deployment. To test whether your planned change causes a blue/green deployment, use the AWS Management Console, an API, or a Python script.

If your changes cause a blue/green deployment, then OpenSearch Service also automatically updates your domain to the latest available software.

Important: You can't apply a new configuration change when there's an existing blue/green deployment in progress. To make multiple configuration updates, include all changes in a single request. If you submit simultaneous changes, then you receive the "A change is already in progress" error message.

Understand how blue/green deployments affect performance

During blue/green deployments your OpenSearch Service cluster is available for incoming search and indexing requests.

However, you might experience the following performance issues:

• Temporary increase in usage on leader nodes as clusters have more nodes to manage.
• Increased search and indexing latency as OpenSearch Service copies data from previous nodes to new nodes.
• Increased rejections for incoming requests as the cluster load increases during blue/green deployments.

To avoid latency issues and request rejections, it's a best practice to run blue/green deployments when the cluster is healthy and there's low network traffic. To avoid data loss during blue/green deployments, follow the operational best practices for OpenSearch Service.

Monitor the blue/green deployment

Check your AWS CloudTrail Event history to monitor the API calls that OpenSearch Service makes as part of the blue/green deployment. If your OpenSearch Service domain uses fine-grained access control, then you can activate audit logs to track user activity on your OpenSearch Service clusters. OpenSearch Service publishes audit logs to Amazon CloudWatch Logs.

Check notifications in the OpenSearch Service console for information about the performance and health of your domains. OpenSearch Service notifies you about service software updates, Auto-Tune enhancements, cluster health events, and domain errors.

There's no defined time on how long a blue/green deployment can take. The duration depends on multiple factors such as the cluster size, workload, shard size, and shard count. To check the progress of the configuration change stages, check the Domain status on the OpenSearch Service console, or use the DescribeDomainChangeProgress API.

You can use the cat recovery API to monitor the status of your shard relocation. For more information, see CAT recovery API on the OpenSearch website.

To see the shards that OpenSearch Service still must reallocate, run the following command:

curl -X GET "https://end_point/_cat/recovery?v=true&pretty" | awk '/peer/ {print $1" "$2" "$3" "$4" "$18}' | grep -v 100\.0\%

Note: Replace end_point with your domain endpoint.

To list the shard relocation by byte percentages, run the following command:

curl -X GET "https://end_point/_cat/recovery?v=true&pretty" | awk '/peer/ {print $1" "$2" "$3" "$4" "$18}' | tr -d "%" | sort -k 5 -n

Note: Replace end_point with your domain endpoint.

If you observe minimal progress for the shard relocation, then your cluster might be stuck.

Troubleshoot stuck blue/green deployments

Your blue/green deployment process might get stuck for the following reasons:

• An cluster state is unhealthy before the configuration change.
• Your Java Virtual Machine (JVM) memory pressure is consistently high.
• Your CPU usage is consistently high.
• You didn't choose the right number or size of shards for the cluster.
• The configuration setup isn't valid, or you made a configuration change when another one was already in progress.
• There's insufficient disk space or capacity for the relocation process or requested Amazon Elastic Compute Cloud (Amazon EC2) instance type.
• There aren't enough available IP addresses on the requested subnet for a cluster inside a virtual private cloud (VPC).
• You used an Amazon Elastic Block Store (Amazon EBS) volume that's too large for the EC2 instance type.
• The index has a write block. To resolve this issue, remove index.routing.allocation.require._name, NODE_NAME, and index.blocks.write: true from your index settings.

To troubleshoot further, see Why is my OpenSearch Service domain stuck in the "Modifying" state?

For clusters that are stuck in the Modifying state or domains that are stuck in the Deleting older resources state for over 24 hours, create an AWS Support case.

Cancel a blue/green deployment

To undo a configuration change, wait for the blue/green deployment to complete. Then, change your configuration back to its previous settings. You can't cancel or pause a blue/green deployment that's in progress.

Related information

Why is my OpenSearch Service domain upgrade taking so long?

Introducing Auto-Tune in OpenSearch Service