How do I troubleshoot an UltraWarm storage migration issue in OpenSearch Service?

Resolution

When you migrate an index to UltraWarm storage in OpenSearch Service, you might receive the following error message: "Rejecting migration request for index [index_name] since there isn't enough disk space on hot node [qwertpoiy123lke4woj34n] to perform force merge".

Before an UltraWarm storage migration, OpenSearch Service validates the amount of available hot disk space. OpenSearch Service automatically runs a force merge operation during the disk space validation. The force merge operation processes one shard at a time. Each shard requires three times the amount of occupied hot disk space and an additional 20 GB of free storage space for each node. If OpenSearch Service detects low disk space, then it blocks the force merge operation request.

Before you migrate an index, complete the following steps:

1. To check for nodes that have shards in the index, run the following command:

   GET _cat/shards/INDEXNAME?v

   Note: Replace INDEXNAME with your OpenSearch Service index.
2. To check the available disk space for each node, run the following command:

   GET /_cat/allocation?v

To calculate your storage requirements, see Calculating storage requirements.

To resolve low disk space issues, use the following best practices:

• Delete old or unused documents from your index. Too many documents can increase shard size and slow down a migration.
• If you can't delete indices from the node, then increase the Amazon Elastic Block Store (Amazon EBS) volume. For more information, see How do I troubleshoot low storage space in my OpenSearch Service domain?
  Note: When you delete a document, OpenSearch Service doesn't immediately free up disk space. For more information, see Why is the Deleted Documents metric so high in my Amazon OpenSearch Service cluster?
• Monitor disk utilization metrics, such as FreeStorageSpace, in Amazon CloudWatch. For more information, see ClusterBlockException.
• Make sure that you have enough storage space in your UltraWarm nodes. To calculate UltraWarm storage requirements, use only the size of the primary shards.