Open Search Mster instance connetivity issues
We are seeing daily 2-3 minute connectivity drops with our OpenSearch service. Have tried to the best of our abilities to troubleshoot this but turning on logs for application and errors seems to log nothing regarding this and to the best of our abilities there does not seem to be resource limitations, we are not using any heavy loads.
In Cloudwatch we can see in the dashboard for "Mster instance connection status" that several times daily there is a 2 minute drop at different times, which feels like some sort of disconnect or reboot of some kind. But we are unable to locate what it is or why, any help on how to troubleshoot further would be greatly appreciated.
- Newest
- Most votes
- Most comments
Based on your description of daily 2-3 minute connectivity drops with your OpenSearch service and the CloudWatch dashboard showing "Master instance connection status" drops, you're likely experiencing an issue with the master node becoming unreachable.
This type of issue is specifically monitored by the MasterReachableFromNode CloudWatch metric, which indicates when the master node stops or becomes unreachable. These failures are typically the result of network connectivity issues or an AWS dependency problem.
To troubleshoot this further:
-
Monitor the
MasterReachableFromNodemetric: Set up an alarm if the maximum value is less than 1 for 5 minutes, as this confirms master node reachability problems. -
Analyze logs and monitoring data: Look for patterns or events that coincide with these connectivity drops.
-
Check for known errors: Review AWS documentation and OpenSearch release notes to see if there are any known issues that might be causing these periodic disconnections.
-
Verify configuration: Review your OpenSearch configuration, including IAM policies, user permissions, and network settings to ensure everything is properly set up.
-
Review cluster architecture: Confirm that you're following best practices for shard and cluster architecture, as improper configuration can lead to instability.
If these steps don't resolve the issue, I recommend reaching out to AWS Support with details about your domain configuration, the specific times when these drops occur, and any patterns you've observed. They can provide more targeted assistance based on the specific characteristics of your OpenSearch Service domain.
Sources
AOSOPS04-BP01 Train staff on common OpenSearch issues and how to remediate them - Amazon OpenSearch Service Lens
Recommended CloudWatch alarms for Amazon OpenSearch Service - Amazon OpenSearch Service
answered a year ago
Relevant content
- AWS OFFICIALUpdated 4 months ago
