How do I create an index pattern in my Amazon OpenSearch Service cluster?

Short description

To create an index pattern, your cluster must meet the following prerequisites:

1. The user must have PUT and POST permissions to create index pattern similar to the following access policy:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "VisualEditor0",
         "Effect": "Allow",
         "Action": [
           "es:ESHttpHead",
           "es:ESHttpPost",
           "es:ESHttpGet",
           "es:ESHttpDelete",
           "es:ESHttpPut"
         ],
         "Resource": "arn:aws:es:<region>:<account-id>:domain/<domain-name>/*"
       }
     ]
   }

2. Your cluster version supports index pattern.

Resolution

Follow these instructions to create an index pattern for OpenSearch Service or Elasticsearch clusters with or without fine-grained access control (FGAC).

Create index pattern for clusters without FGAC activated

Elasticsearch clusters

Run a curl command similar to the following:

curl -X POST  https://elasticsearch-end-point/_plugin/kibana/api/saved_objects/index-pattern/ \

-H "kbn-xsrf: true" \

-H "content-type: application/json" \

-d '{ "attributes": { "title": "sample-index*" } }'

OpenSearch Service clusters

Run a curl command similar to the following:

curl -X POST  https://opensearch-end-point/_dashboards/api/saved_objects/index-pattern/ \

-H "kbn-xsrf: true" \

-H "content-type: application/json" \

-d '{ "attributes": { "title": "sample-index*" } }'

Create index pattern for clusters with FGAC activated

Elasticsearch clusters

Run the following curl command to generate authorization cookies into the auth.txt file:

curl -X POST  https://elasticsearch-end-point/_plugin/kibana/auth/login  \
-H "kbn-xsrf: true" \
-H "content-type: application/json" \
-d '{"username":"<username>", "password":"<password>"}' \
-c auth.txt

Then, submit the index pattern creation request:

curl -X POST  https://elasticsearch-end-point/_plugin/kibana/api/saved_objects/index-pattern/test  \
-H "kbn-xsrf: true" \
-H "content-type: application/json" \
-d '{ "attributes": { "title": "sample-index*" } }' \
-b auth.txt

OpenSearch Service clusters

Run the following curl command to generate authorization cookies into the auth.txt file:

curl -X POST  https://opensearch-end-point/_dashboards/auth/login  \
-H "kbn-xsrf: true" \
-H "content-type: application/json" \
-d '{"username":"<username>", "password":"<password>"}' \
-c auth.txt

Then, submit the index pattern creation request:

curl -X POST  https://opensearch-end-point/_dashboards/api/saved_objects/index-pattern/  \
-H "kbn-xsrf: true" \
-H "content-type: application/json" \
-d '{ "attributes": { "title": "sample-index*" } }' \
-b auth.txt

(Optional) Python client

Use the following python code snippet for creating the index pattern:

import boto3
import requests
from requests_aws4auth import AWS4Auth

host = 'https://domain-endpoint/' # include trailing /
region = 'es-region' # example us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)


path = '_dashboards/api/saved_objects/index-pattern' # _plugin/kibana/api/saved_objects/index-pattern for es versions
url = host + path
payload = {"attributes":{"title":"multi-logs-*","fields":"[]"}}
headers = {"Content-Type": "application/json", "osd-xsrf": "true", "security_tenant": "global" }
r = requests.post (url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)

Note: You must have the following dependencies installed:

pip install boto3
pip install opensearch-py
pip install requests
pip install requests-aws4auth

Note:

• The role used to run the code snippet must map to the backed role for FGAC clusters.
• For Elasticsearch clusters, change the path to '_plugin/kibana/api/saved_objects/index-pattern'.

Troubleshoot common issues with creating an index pattern

FGAC clusters using SAML or Amazon Cognito authentication

If the domain for your cluster uses SAML or Amazon Cognito for authentication, then create an internal user to manage the index pattern.

Note: The user must have PUT and POST permissions to create an index pattern. This requirement isn't required for clusters without FGAC activated.

Create an index pattern in other tenants

Index patterns are created under the Global tenant by default. To create an index pattern outside of the Global tenant, run a curl command similar to the following:

curl -s -X POST https://opensearch-end-point/_dashboards/api/saved_objects/index-pattern/sample-index -d '{"attributes": {"title": "sample-index*"}}' \
-H "osd-xsrf:true" \
-H "securitytenant: private" \
-H "content-type:application/json" \
-b auth.txt

Missing .kibana alias in the cluster

Check if the .kibana alias exists without FGAC activated in the cluster using the following curl command:

curl -XGET https://opensearch-end-point/_cat/aliases

Check if the .kibana index exists with FGAC activated in the cluster using the following curl command:

curl -XGET https://opensearch-end-point/_cat/aliases

If the .kibana index doesn't exist, skip steps 1-2 to create a backup and delete the index.

1. Create a backup of .kibana index:

curl -XPOST "https://domain-end-point/_reindex" -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": ".kibana"
  },
  "dest": {
 "index": ".kibana_backup"
  }
}'

2. Delete the .kibana index:

curl -XDELETE "https://domain-end-point/.kibana"

3. Create a .kibana alias and point it to the .kibana_backup index:

curl -XPOST "https://domain-end-point/_aliases" -H 'Content-Type: application/json' -d'
{
  "actions": [
    {
      "add": {
        "index": ".kibana_backup",
        "alias": ".kibana"
      }
    }
  ]
}'

Note: Include the authentication (username:password) parameter for FGAC clusters.

Related information

Export and import Kibana dashboards with Amazon ES

Why does the rollover index action in my ISM policy keep failing in OpenSearch Service?