How do I troubleshoot Kibana errors in OpenSearch Service?

Short description

Kibana or OS dashboards are visualization plugins that are installed on each data node of a cluster. These plugins sit on top of the OS or ES process and use the same data node resources (CPUUtilization and JVMPressure).

The KibanaHealthyNodes metric or Kibana Health status in the AWS Management Console shows the current state of the Kibana plugin running on your data nodes:

• Green - Kibana is running on all data nodes and there are no known issues.
• Yellow - Kibana is running on some, but not all, of the data nodes. Kibana runs slowly in this status.
• Red - Kibana is down on all nodes. Kibana doesn't run in this status and users don't have access to Kibana.

When you have issues with Kibana, you might receive one of the following errors:

• Kibana blank page/ unable to see any data on Kibana dashboard
• Kibana server is not ready yet
• Kibana did not load properly (when you try to access the Kibana URL provided by Amazon OpenSearch Service)
• Kibana did not load properly. Check the server output for more information.

Resolution

To troubleshoot the Kibana errors that you receive, check for the following:

• Check that your CPU utilization doesn't exceed 80%. For more information, see How do I troubleshoot high CPU utilization on my Amazon OpenSearch Service cluster?
• Check that your JVM memory pressure metric doesn't exceed 75%. For more information, see How do I troubleshoot high JVM memory pressure on my OpenSearch Service cluster?
• When your JVM memory pressure metric exceeds 92% for 30 minutes, the OpenSearch Service activates a protection mechanism, and then blocks all write operations. This action is done to prevent the cluster from reaching a red status. During this protection, write operations fail with a ClusterBlockException error and new indexes can't be created. As a result, the IndexCreateBlockException error occurs. For more information, see How do I resolve the 403 "index_create_block_exception" or "cluster_block_exception" error in OpenSearch Service?
• Run the _cat/allocation API to check the disk percentage, storage, and sharding strategy of your cluster on each node. If your disk percentage is higher than 80%, then your cluster might underwrite the block state. For more information, see How do I troubleshoot low storage space in my OpenSearch Service domain?
• Make sure that you distribute shards evenly across nodes. Disk percentage can only be high on a few nodes because of skewness. For more information, see How do I rebalance the uneven shard distribution in my Amazon OpenSearch Service cluster?
• Make sure that shards don't exceed the default threshold value of 1000 per node. If they do, then decrease the shard count per node. To decrease the shard count, either delete unwanted indices or add more nodes to your cluster. To delete unwanted indices, see Delete index API on the Elastic website. For more information, see Choosing the number of shards.
• Check for failed nodes. For more information, see Why did my OpenSearch Service node crash?
• Check your queries. Make sure that you don't use bulk queries because doing so creates a heavy load on your cluster. If you must use bulk queries to improve indexing performance, then start with a small bulk size. Then, increase incrementally. For more information, see Using and sizing bulk requests on the Elastic website.
• Check if the domain has an access policy. The OpenSearch Service domain must have an access policy that allows users and roles to interact with the service. For more information, see Types of policies.

Note: If you use a resource-based policy or identity-based policy, then you can't access the Kibana dashboard. If you use these policies, then you must have an AWS Identity and Access Management (IAM) role or user present in the access policy. Then, you must sign your request to connect with the OpenSearch Service domain through API calls. For more information, see Signing HTTP requests to Amazon OpenSearch Service. If you have a public domain, then use IP-based policies to allow access to your domain from specific IP addresses only.

• If you receive a cannot restore index [.kibana] error, then use the _reindex API command to restore the indices and rename the .kibana index. For more information, see How do I resolve the "cannot restore index [.kibana]" error in Amazon OpenSearch Service?

If you still have issues with Kibana, then contact AWS Support.