How do I troubleshoot a “No endpoints available” error in the DAX client?

Short description

The "No endpoints available" error occurs when the client doesn't have a network route to any nodes of the DAX cluster. This error might also occur when a cluster is down or experiences high CPU utilization.

The following are example cases that might result in this error:

• A failed health check to DAX nodes: Because of high CPU load from the nodes, the client can't perform a health check on the cluster nodes.
• Multiple consecutive I/O exceptions to each node: The DAX client counts the number of I/O exceptions to determine whether a server node is down. The client retrieves a list of server nodes from the configuration endpoint. If it receives more than five failures from a server node, then the client removes the node from the list of active server nodes. If all server nodes are removed in this way, then the client ends up with an empty list of active nodes. In response, the client shows a "No endpoints available" error.

Resolution

To resolve the error, complete the following tasks:

• For the AWS Identity and Access Management (IAM) role that the client uses, update the role to set the proper DAX permissions.
• In the DAX client, there's a thread to refresh the list of active server nodes. To restore your active nodes in the list, use the refresh option.
• Increase the RequestTimeout based on the client's traffic pattern. For more information, see Tuning AWS Java SDK HTTP request settings for latency-aware Amazon DynamoDB applications.
• Increase the health check interval. By default, the health-check-interval is 5 seconds.
  Note: Increasing this interval puts the endpoint back into the client-side endpoint set only when it's removed because of a socket-timeout-exception .
• To reduce the reoccurrence of this error, scale up to a larger instance type. Or, add more nodes to the cluster to scale out.
• If your cluster has only one node, then scale out the cluster. Place each node in a different Availability Zone. This error is more likely to occur with single node clusters.

Related information

ClientConfig