내용으로 건너뛰기

Adding snapshot action to ISM

0

We have an OpenSearch cluster running 2.19 which we use to store logs.

We currently delete logs after 10 days.

{
    "policy": {
      "description": "Delete logs after 10 days",
      "default_state": "hot",
      "ism_template": [
        { "index_patterns": ["main-*"], "priority": 100 }
      ],
      "states": [
        {
          "name": "hot",
          "actions": [],
          "transitions": [
            { "state_name": "delete", "conditions": { "min_index_age": "10d" } }
          ]
        },
        {
          "name": "delete",
          "actions": [
            { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "delete": {} }
          ],
          "transitions": []
        }
      ]
    }
  }

This is working as expected. We now want to add a step to back them up to S3 before deleting. We have added the S3 bucket to Repositories in OpenSearch and did a Snapshot to ensure all permissions are working. I can see the .dat files in the S3 bucket.

However, trying to update the policy is causing an issue

{
    "policy": {
      "description": "Send to S3 then delete after 10 days":,
      "default_state": "hot",
      "ism_template": [
        { "index_patterns": ["main-*"], "priority": 100 }
      ],
      "states": [
        {
          "name": "hot",
          "actions": [],
          "transitions": [
            { "state_name": "archive", "conditions": { "min_index_age": "10d" } }
          ]
        },
        {
          "name": "archive",
          "actions": [
            {
"retry": {
"count": 5,
"backoff": "exponential",
"delay": "30m"
},
              "snapshot": {
                "repository": "main-app-logs-bucket",
                "snapshot": "{{ctx.index}}-{{ctx.execution_time}}"
              }
            }
          ],
          "transitions": [
            { "state_name": "delete" }
          ]
        },
        {
          "name": "delete",
          "actions": [
            { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "delete": {} }
          ],
          "transitions": []
        }
      ]
    }
  }

This get's an error saying

You have actions that are not allowed in your policy [snapshot]

질문됨 일 년 전176회 조회

1개 답변
0

The error you're encountering is because the snapshot operation in Index State Management (ISM) has specific version requirements. The snapshot operation is only supported in Amazon OpenSearch Service domains running OpenSearch or Elasticsearch 7.7 or later.

Based on your error message, it appears that your OpenSearch cluster may not meet this version requirement or the snapshot functionality might not be properly enabled for ISM in your environment.

To fix this issue, you should:

  1. Verify that your OpenSearch cluster version supports the snapshot operation in ISM policies.
  2. Ensure that the snapshot repository "main-app-logs-bucket" is properly registered in your OpenSearch domain.
  3. Make sure the IAM role associated with your OpenSearch domain has the necessary permissions to access the S3 bucket.

If you're using Amazon OpenSearch Service, the snapshot repository needs to be registered properly with the correct permissions. This includes:

  • Creating an IAM role with the necessary S3 permissions
  • Setting up a custom S3 bucket for storing snapshots
  • Registering the S3 bucket in your OpenSearch Service domain

Your policy structure looks correct otherwise, with the proper retry mechanism and transitions between states. The snapshot name using the dynamic variables {{ctx.index}}-{{ctx.execution_time}} is also valid and will create unique snapshot names based on the index name and execution time.

As an alternative, you might also consider using Snapshot Management (SM) policies instead of ISM for automating snapshots if that's available in your OpenSearch version. SM policies allow you to specify a schedule for taking snapshots automatically.
Sources
Index State Management in Amazon OpenSearch Service - Amazon OpenSearch Service
Tutorial: Automating Index State Management processes - Amazon OpenSearch Service
AOSOPS02-BP01 Establish a manual snapshot repository - Amazon OpenSearch Service Lens
AOSOPS02-BP02 Automate the process of taking snapshots - Amazon OpenSearch Service Lens

답변함 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

관련 콘텐츠