How do I troubleshoot fine-grained access control issues in my OpenSearch Service cluster?

Resolution

Prerequisites:

• To use other AWS services with OpenSearch Service, make sure that you provide the required permissions to the service AWS Identity and Access Management (IAM) roles.
• To provide fine-grained access control to specific indexes or dashboards, map the user to a role with the following permissions to the tenant's Kibana index:

  .kibana_hash_tenant_name

  Note: Replace tenant_name with your tenant name. OpenSearch automatically generates a value for hash.
• To use fine-grained access control at the field level, set up an OpenSearch security role with the required field-level security. For instructions, see Defining roles on the OpenSearch website. Then, to map the user to the role, see Mapping users to roles on the OpenSearch website. For more information, see Field-level security on the OpenSearch website.
• To use fine-grained access control at the document level, create an OpenSearch security role with the required document-level security. Then, to map the user to the role, see Mapping users to roles on the OpenSearch website. For more information, see Document-level security on the OpenSearch website.

If you use fine-grained access control and receive errors, then take the following troubleshooting actions based on the error that you receive.

Note: You can't use anonymous access in OpenSearch Service.

"security_exception","reason":"no permissions for [action]" 403 error

To resolve this error, make sure that the user or backend role in your OpenSearch Service cluster has the required permissions. For information about required permissions, see Permissions on the OpenSearch website. Then, make sure that you mapped the user or backend role to an OpenSearch security role. For instructions, see Mapping users to roles on the OpenSearch website.

"security_exception","reason":"no permissions for []" error

You must exclude system indexes from restore requests. If you don't, then you might encounter this error when you restore a snapshot, even if you have the required permissions.

To resolve this issue, run the following command to exclude system indexes from the restore request:

curl -XPOST 'domain-endpoint/_snapshot/snapshot-repository/snapshot-name/_restore' \
-d '{"indices": "-.kibana*,-.opendistro*"}' \
-H 'Content-Type: application/json'

Note: Replace domain-endpoint with your domain endpoint, snapshot-repository with your snapshot repository, and snapshot-name with your snapshot name.

"Missing role" error

To log in to OpenSearch Dashboards, you must map the user or backend role to at least one OpenSearch security role. If you don't, then you might receive the following error message when you log in to OpenSearch Dashboards:

"Missing role. No roles available for this user, please contact your system administrator."

To resolve this issue, see Mapping users to roles on the OpenSearch website.

"User: anonymous is not authorized to perform: iam:PassRole" error

You must map the manage_snapshots role to the IAM role that you used to register the manual snapshot. If you don't, then you might receive this error when you register a manual snapshot repository.

To resolve this issue, complete the following steps:

1. Log in to your OpenSearch Dashboard as the administrative user.
2. In the navigation pane, choose Security.
3. Choose Roles.
4. Select manage_snapshots.
5. Choose Mapped Users, and then choose Manage Mapping.
6. For Backend Roles, enter the IAM role's Amazon Resource Name (ARN).
7. Press Enter.
8. Choose Map.

"User: anonymous is not authorized to perform: es:ESHttp HTTP_Method" error

The requester must adhere to the domain access policy requirements. If they don't, then they receive the "not authorized to perform" error message.

To resolve this error, update the domain access policy to allow the type of request that you send. To allow unsigned requests, set Principal to "AWS": "*".

Then, take the following actions based on the requirements of your policy.

The policy includes an IAM principal

If the domain access policy includes an IAM principal, then the IAM principal must sign the request with AWS Signature Version 4. In this scenario, OpenSearch Service rejects unsigned requests, such as basic authorization or anonymous access.

The following example command shows a request that the IAM role signs:

curl -X GET "opensearch_endpoint" \
      --user "AWS_ACCESS_KEY_ID":"AWS_SECRET_ACCESS_KEY" \
      -H "x-amz-security-token: AWS_SESSION_TOKEN" \
      --aws-sigv4 "aws:amz:region:es"

Note: Replace opensearch_endpoint with your endpoint, AWS_ACCESS_KEY_ID with your access key, AWS_SECRET_ACCESS_KEY with your secret access key, AWS_SESSION_TOKEN with your session token, and region with your AWS Region. For requests from an IAM user, remove -H "x-amz-security-token: AWS_SESSION_TOKEN" \.

The policy allows requests only from specific IP addresses

If the policy has IP address conditions and uses a "Principal": { "AWS": "*" } open principal, then you can use unsigned requests to access OpenSearch Service. However, the request must come from an allowed IP address.

Note: If the policy includes an IAM principal and only allows a specific IP address, then the IAM principal must sign the request. Also, the request must be from an allowed IP address.

The policy blocks specific HTTP actions

Make sure that the domain access policy allows the HTTP method that you request. For example, if you send a POST request, then make sure to include the es:ESHttpPost action in the policy.

"Couldn't find any Elasticsearch data" error

You might receive this error when you create index patterns in a cluster with fine-grained access control after you upgrade to OpenSearch Service version 7.9. Make sure that the role assigned to the user has the indices:admin/resolve/index permission on the relevant indexes and aliases. For more information, see Resolve Index API on the OpenSearch website.

"401 unauthorized" errors

To access a cluster with fine-grained access control, you must use the -u flag to authenticate. For example, the following command fails with a "401 unauthorized" error message:

curl Domain_Endpoint

To access your cluster, run the following command and use the -u flag to include your username and password:

curl -u 'username:password' Domain_Endpoint

Note: Replace username with your username, password with your password, and Domain_Endpoint with your domain endpoint.

You must use valid characters when you run curl commands. If you use the $ or ! characters in primary credentials with the -u flag in curl commands, then you might receive "401 unauthorized" errors. Also, make sure to put your credentials in single quotes.

Example command:

curl -u 'username' Domain_Endpoint

Note: Replace username with your username, and Domain_Endpoint with your domain endpoint.

"Authentication finally failed" error

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

You must send a request that matches your domain's user type. If you send a basic (username/password) authentication request, but you configured the domain's master user type as an IAM principal, then you receive this error. With a master user, your client must send a request that's signed by AWS Signature Version 4 to the domain.

To use HTTP basic authentication, run the following update-domain-config AWS CLI command to change the master user type to an internal user database:

aws opensearch update-domain-config --domain-name domain-name --advanced-security-options InternalUserDatabaseEnabled=true

Note: Replace domain-name with your domain name.

Related information

Index Management on the OpenSearch website