How do I troubleshoot OpenSearch Serverless Dashboards access errors?

Short description

Resources that are in the same Amazon Virtual Private Cloud (Amazon VPC) as the Amazon VPC endpoint can access OpenSearch Dashboards. Or, connect to the VPN for the VPC to access OpenSearch Dashboards.

If you still can't access OpenSearch Dashboards, then one of the following issues prevents access to OpenSearch Dashboards:

• Your connection times out because your resource doesn't have access to the VPC endpoint.
• You receive a "401 Unauthorized" error because your client machine is outside the VPC, or an access policy is denying access.
• You receive an "HTTP Error 403: You don't have authorization to view this page" error because you don't have permission to access OpenSearch Dashboards.

Resolution

Prerequisite: Open the OpenSearch Service console, and note your VPC endpoint URL. See the following example URL:

https://0123456789abcdefghij.ap-southeast-2.aoss.amazonaws.com

Resolve connection timeout errors

If your connection times out when you access OpenSearch Dashboards from your browser, then the resource can't access the VPC endpoint. You must add the source security group or IP address to the inbound rules of security group that's associated with the OpenSearch Serverless VPC endpoint.

Resolve the DNS host of the VPC endpoint

Complete the following steps:

1. On the client machine, open the terminal and run the following command:

   nslookup example-endpoint-URL

   Note: Replace example-endpoint-URL with your VPC endpoint URL.
   In the output, note the IP address that you must add to the inbound rules of your security group. See the following example output:

   Non-authoritative answer:
   Name:  privatelink.a00.b11.cde.prod.aoss.searchservices.aws.dev
   Addresses: 172.X1.Y1.123
        172.X2.Y2.456
   Aliases: 0123456789abcdefghij.us-east-1.aoss.amazonaws.com

2. To run a telnet test on port 443, run the following command:

   telnet example-IP-address 443

   Note: Replace example-IP-address with the IP address from the output that you received.

3. In the output, check that you can connect to the VPC endpoint. The following example output shows that you can connect to the endpoint:

   $ telnet 172.X1.Y1.123 443
   Trying 172.X1.Y1.123...
   Connected to 172.X1.Y1.123.
   Escape character is '^]'.

Use Reachability Analyzer to identify if Amazon EC2 can connect to the VPC endpoint

If you use Amazon Elastic Compute Cloud (Amazon EC2) to access your VPC endpoint, then use Reachability Analyzer to troubleshoot connectivity issues.

Complete the following steps:

1. Open the AWS Network Manager console.
2. In the navigation pane, under Monitoring and troubleshooting, choose Reachability Analyzer.
3. Choose Create and analyze path.
4. On the Create and analyze path page, enter the following information: Under Path source, for Source type, choose Instances. Then, select your Amazon EC2 instance. Under Path destination, for Destination type, choose VPC endpoints. Then, select the collection's VPC endpoint. For Protocol, choose TCP. Choose Create and analyze path.
5. Under the Analyses tab, review the reachability test results.

Resolve "401 Unauthorized" errors

The following issues cause"401 Unauthorized" errors:

• Your client machine has an IP address that's included in the security group inbound rules, but the machine is outside the VPC.
• A network access policy denies you access to the collection.

Resolve the DNS host of the VPC endpoint

Complete the following steps:

1. On the client machine, open the terminal and run the following command to check that the hostname resolves to a private link:

   nslookup example-endpoint-URL

   Note: Replace example-endpoint-URL with the VPC endpoint URL.

2. If your DNS resolves to a public hostname, then review the configurations for your VPC, subnet, and security group. See the following example DNS resolution:

   Server: ip-A1-B-C2-D.ap-southeast-2.compute.internal
   Address: 10.A.BC.D
   
   Non-authoritative answer:
   Name:  example.sgw.syd.prod.aoss.searchservices.aws.dev
   Addresses: 3.X1.YZ1.55
        54.X2.YZ2.95
        54.X3.YZ3.119
   Aliases: c57qhobw71y368nmhkkc.ap-southeast-2.aoss.amazonaws.com

3. Review the resolver host and IP address configuration and check for unexpected resolvers, such as a proxy resolver.
   Note: If the configuration is an unexpected resolver, then the resolver internally uses another Amazon VPC resolver.

4. If nslookup resolves to a private link and you continue to get 401 errors, then create a HAR file to troubleshoot the error.

Create a HAR file

Reproduce the issue in your browser, and then create a HAR file to determine the cause of the error.

In the following example HAR file, the X01:network-policy-deny network access policy denies access to OpenSearch Dashboards:

"response": 
{     
  "status": 401,
  "statusText": "Unauthorized",
  "httpVersion": "HTTP/1.1",
  "headers": [      
    {       
      "name": "content-length",
      "value": "0"      
    },
    {       
      "name": "date",
      "value": "Thu, 30 Mar 2023 00:29:21 GMT"
    },      
    {       
      "name": "server",       
      "value": "aoss-amazon"      
    },      
    {       
      "name": "x-aoss-response-hint",
      "value": "X01:network-policy-deny"
    },
    {       
      "name": "x-request-id",
      "value": "b1211888-1234-9e64-9999-aaxyzab1fd6"
    }
  ],
  ...

To resolve this issue, update your network access policy so that the VPC endpoint in the network policy matches your collection's VPC endpoint.

Resolve "HTTP ERROR 403" errors

"HTTP ERROR 403" error messages show that the user profile doesn't have permission to access OpenSearch Dashboards. Reproduce the issue in your browser. Then, create a HAR file to determine the required permissions to access OpenSearch Dashboards.

Example HAR file:

"response": 
{     
  "status": 403,
  "statusText": "Forbidden",
  "httpVersion": "HTTP/1.1",     
  "headers": [      
    {
      "name": "content-length",
      "value": "0"      
    },
    {       
      "name": "date",
      "value": "Mon, 17 Apr 2023 00:10:25 GMT"
    },      
    {       
      "name": "server",       
      "value": "aoss-amazon-d"      
    },
    {       
      "name": "x-aoss-response-hint",
      "value": "X01:dashboards-authz-denied"      
    },      
    {       
      "name": "x-envoy-upstream-service-time",
      "value": "19"      
    },      
    {       
      "name": "x-request-id",
      "value": "b559fd8f-315e-9fe9-a9e8-6ff5791b765a"
    }
    ...

Update your AWS Identity and Access Management (IAM) permissions to include the required permissions.

Additional troubleshooting steps

Update your IAM permissions for data plane policies

To access OpenSearch Serverless data plane API operations and OpenSearch Dashboards from a browser, update your IAM permissions. Be sure to add the aoss:APIAccessAll and aoss:DashboardsAccessAll IAM permissions to the permissions policy.

Update your IAM user or group configurations

Update the data access policy that's attached to your collection to include the correct permissions for the IAM user or group. Then, sign in to the AWS Management Console as the IAM user or group that has the data access policy permissions. OpenSearch Dashboards automatically uses the IAM credentials to log you in to OpenSearch Dashboards.

Update your SAML user or group configurations

For SAML users or groups, sign in with entity that has the correct data access policy permissions. Update the user or group to match the SAML provider configuration.

Note: The mappings between the user or group and the SAML provider are case sensitive and must match.

To troubleshoot the contents of real assertations, use the SAML-tracer. For more information, see Configure SAML fields.

Don't open the OpenSearch Dashboards URL from Management Console. Instead enter the URL in a new tab or window. If you use the link from Management Console, then the console tries to authenticate the IAM user.

Related information

How do I set up an OpenSearch Serverless collection with an Amazon VPC endpoint and access the collection's dashboard?