Amazon MQ "socket.gaierror: [Errno -2] Name or service not known"

Question

Suddenly today our application failed health checks. In our application logs we see: "socket.gaierror: [Errno -2] Name or service not known". We use the fully qualified DNS name to connect, e.g. abc-123.us-east-1.amazonaws.com.

In CloudWatch for Amazon MQ, I see some references to "RabbitMQ is asked to stop..." followed by BEAM shutting down various processes within the supervisor tree.

CPU and memory load on our Amazon MQ instance look completely fine. We also do **not** allow minor software upgrades.

As best I can tell, my hunch is there was a DNS issue, or some kind of Amazon-triggered configuration change that temporarily took us out of service. (I'll note that we are not using a cluster, but rather a single instance.)

I would really love to be able to explain what happened and why it happened, but unfortunately I feel like I don't have quite the full story.

Answer

Hello,

The error "socket.gaierror: [Errno -2] Name or service not known" typically points to a DNS resolution issue. This error indicates that the hostname used in your application cannot be resolved to an IP address.

Here are some steps to troubleshoot and resolve the issue:

1. DNS Configuration: Double-check the DNS name you are using in your application. Ensure it is correctly configured and resolves to the correct IP address.

2. Check Network Connectivity: Verify that your application server has network connectivity to the Amazon MQ broker. Test basic connectivity using tools like `ping` or `telnet` to ensure there are no network issues.

3. Review Security Groups and Network ACLs: Ensure that the security groups and network ACLs associated with your Amazon MQ broker allow the necessary inbound and outbound traffic. This includes DNS resolution and communication with the broker.

4. DNS Resolution from Application Server: Use tools like `nslookup` or `dig` on the machine where your application is running to verify that DNS resolution is working correctly. Check if the specified DNS name can be resolved to an IP address.

5. Amazon MQ Endpoint: Confirm that the Amazon MQ broker's endpoint is correctly specified in your application. You can find the endpoint in the Amazon MQ console.

6. Temporary DNS Issues: Sometimes, DNS issues can be transient. If the error occurred briefly, it might be worth retesting to see if the DNS resolution issue persists.

7. Restart Application and Broker: If all configurations seem correct, try restarting your application and the Amazon MQ broker to see if the issue is resolved.

8. Check Amazon MQ Metrics: Review Amazon MQ metrics in CloudWatch for any anomalies or issues related to connections or DNS resolution.

Amazon MQ "socket.gaierror: [Errno -2] Name or service not known"

Relevant content