How do I use ISM to manage low storage space in Amazon OpenSearch Service?

Short description

To resolve low storage space, use ISM to define custom management policies and perform a rollover operation, such as to automatically delete old indices. The rollover operation rolls over a target to a new index when an existing index meets specified conditions. ISM evaluates the index creation date and time at each state to determine when to start the associated actions.

After you attach your policy to an index, your index begins to initialize and then transitions into different states until the rollover operation completes. For more information about the rollover operation, see rollover on the Open Distro website.

Resolution

Set up your rollover index

To create an index pattern in OpenSearch Dashboards, complete the following steps:

1. Open the OpenSearch Service console.
2. Access OpenSearch Dashboards.
   Note: You can find the link to OpenSearch Dashboards in the domain summary of your OpenSearch service console.
3. Choose Management.
4. Choose Index Patterns, and then choose Create index pattern.
5. Create an index and alias with an index format that matches the ^.*-\d+$ index pattern.
   Important: To prevent an error, make sure that you correctly configure your rollover alias.

For more information, see Index patterns on the OpenSearch website.

When you include a rollover operation in the ISM policy, you must also include a rollover alias. For more information, see Why does the rollover index action in my ISM policy keep failing in OpenSearch Service?

The following example creates test-index-000001 and populates it with several documents. Because the example uses a new index that the target rolled over to, the index format must match the pattern of the index:

PUT test-index-000001/_doc/1
{
  "user": "testuser",
  "post_date": "2020-05-08T14:12:12",
  "message": "ISM testing"
}

The following example creates the test-index alias that points toward your new index:

POST /_aliases{
  "actions": [
    {
      "add": {
        "index": "test-index-000001",
        "alias": "test-index"
      }
    }
  ]
}

Create an ISM policy

In OpenSearch Dashboards, create an ISM policy for your rollover operation.

Example policies

Roll over to warm state:

{  "policy": {
    "policy_id": "Roll_over_policy",
    "description": "A test policy. DO NOT USE FOR PRODUCTION!",
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_size": "10mb"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm"
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "replica_count": {
              "number_of_replicas": 2
            }
          }
        ],
        "transitions": []
      }
    ]
  }
}

The preceding ISM policy defines the hot and warm states. By default, your index is in the hot state. The index transitions into the warm state when the size of the index reaches 10 MB and ISM rolls over to a new index. In the warm state, you can perform different actions on the index. For example, you can change the replica count to two or perform a force_merge operation.

Roll over to delete index after few days:

{  "policy": {
    "policy_id": "Roll_over_policy",
    "description": "A test policy. DO NOT USE FOR PRODUCTION!",
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_size": "10mb"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "30d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ],
        "transitions": []
      }
    ]
  }
}

The preceding ISM policy defines the hot and delete states. By default, the index is in the hot state. After the index reaches 10 MB, ISM rolls over to a new index. Then, after 30 days the index transitions to the delete state and ISM deletes the index.

For more example policies, see Sample policies.

Attach the policy to an index

Complete the following steps:

1. Open the OpenSearch Service console.
2. Use the link in the domain summary of the OpenSearch service console to access OpenSearch Dashboards.
3. Choose the Index Management tab.
4. Select the index that you want to attach your ISM policy to, for example test-index-000001.
   Note: You can find your index under Policy Managed Indices.
5. Choose Apply policy.
6. (Optional) If your policy specifies actions that require an alias, provide the alias, and then choose Apply.

You can also use the following API to attach the ISM policy:

POST _plugins/_ism/add/index_name
{ "policy_id": "policy_name"}

Note: Replace index_name with the name of your index and policy_name with the name of your policy.

For more information, see Add policy on the OpenSearch website.

Update the policy of an existing index

ISM policies in OpenSearch use versioning. When you update a policy, newly created indices automatically reference the latest version of the policy. However, indices that you create before the update continue to reference the previous version. To apply the updated policy to existing indices, you must first remove the old policy from the indices and then reapply the updated version.

To reapply your ISM policy to an existing index, complete the following steps:

1. Open the OpenSearch Service console.
2. Use the link in the domain summary of the OpenSearch service console to access OpenSearch Dashboards.
3. Choose the Index Management tab.
4. In the Policy Managed Indices section, choose Change Policy.
5. Select the indices that you want to apply the change to, for example test-index-000001.
6. Choose the current state of the indices.
7. In the Choose New Policy section, choose Update policy name.
8. (Optional) To change the indices to another state after you update the policy, choose Switch indices to the following state after the policy takes effect. Then, choose the state on the dropdown list.

You can also use the following API to update the ISM policy:

POST _plugins/_ism/change_policy/indexname
{ "policy_id": "policy_1"}

Note: Replace index_name with the name of your index and policy_1 with the name of your policy.

For more information, see Update managed index policy on the OpenSearch website.

Add a template to attach the policy to multiple indices

To attach a policy to multiple indices that match the index pattern, create and use an ISM template.

Use the following API call to create the policy:

PUT _plugins/_ism/policies/test_policy

Example policy:

{ 
 "policy": { 
  "description": "A test policy. DO NOT USE FOR PRODUCTION!", 
  "last_updated_time": 1642027350875, 
  "schema_version": 1, 
  "error_notification": null, 
  "default_state": "hot", 
  "states": [ 
    { 
      "name": "hot", 
      "actions": [ 
        { 
          "rollover":{ 
            "min_size": "10mb" 
          } 
        } 
      ], 
      "transitions":[ 
        { 
          "state_name": "warm" 
        } 
      ] 
    }, 
    { 
      "name": "warm", 
      "actions": [
        { 
          "replica_count":{ 
          "number_of_replicas": 2 
        } 
      } 
    ], 
    "transitions": [] 
      } 
    ], 
        "ism_template":{ 
        "index_patterns":[ 
        "test*" 
    ], 
        "priority": 100 
   } 
  } 
}

Use the following example API call to verify that the test_policy template is attached to the newly created index:

GET _plugins/_ism/explain/test-index-000002

Expected output:

{
  "test-index-000002": {
    "index.plugins.index_state_management.policy_id": "test_policy",
    "index.opendistro.index_state_management.policy_id": "test_policy",
    "index": "test-index-000002",
    "index_uuid": "CZrQ-RzRS8SmiWIuyqFmVg",
    "policy_id": "test_policy",
    "enabled": true
  },
  "total_managed_indices": 1
}

For more information, see Explain index on the OpenSearch website.

Note: The index also populates in the Managed Indices section on the OpenSearch Dashboards Index Management tab.

Check your ISM policy states

For a rollover operation, an index is complete after the index rolls over, transitions into a warm state, and the replica count is updated.

If you use an ISM policy and the index doesn't correctly migrate, then check the status of the ISM.

To check the status of the migration for an index, use the following API:

GET _ultrawarm/migration/index_name/_status

Note: Replace index_name with the name of the index that you're checking the status of.

To get the migration status of all indices, use the following API:

GET _ultrawarm/migration/_status?

Note: To move data from hot storage to warm or cold storage, turn on warm and cold storage on the domain before you configure the ISM policy. The ISM policy must also explicitly include the migration action to transition an index between storage tiers. Without the migration action, only the policy state name changes. The index continues to reside in hot storage.

Configure a warm migration action:

{
        "name": "warm",
        "actions": [{
          "warm_migration": {},
          "retry": {
            "count": 5,
            "delay": "1h"
          }

Configure a cold migration action:

{
        "name": "cold",
        "actions": [{
            "cold_migration": {
              "timestamp_field": "<your timestamp field>"
            }
          }