AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
AWS MQ (Rabbit Engine) Cluster: clients get occasionally `ConnectionResetError 'Connection reset by peer'`
We run a 3 node RabbitMQ Cluster. We use aiopika to connect to the Cluster (link to clients' code). Once in a while some containers would restart because RabbitMQ connection breaks Server connection reset: ConnectionResetError(104, 'Connection reset by peer') and our client does not recover (until container starts new). This happens to many containers but at different times and approximate once every couple days.
QUESTION: What can cause this problem? How to further debug the problem?
Findings
In our cloudwatch logs /connection group we are not able to find anything related to missing heartbeats (so it should not be a missing heartbeat issue?). There are no errors that would hint at anything going wrong. But we have warnings (that also happen on RabbitMQ cluster that we run on our own. These warnings do not seem to be an issue because on our self-managed RabbitMQ Cluster we don't have breaking client connections (ConnectionReset errors). Also these warnings happen much more often compared to when our containers restart because of Connection Reset
2025-11-01 12:40:25.612205+00:00 [warning] <0.25094872.0> client unexpectedly closed TCP connection
2025-11-01 12:40:30.907803+00:00 [warning] <0.25095181.0> closing AMQP connection <0.25095181.0> (<redacted>:32803 -> <redacted>:5671, vhost: '/', user: '<redacted>'):
The only related (to ConnectionReset error) log we get is below. It is not clear why connection is closed.
2025-11-03 07:16:30.172908+00:00 [warning] <0.16537378.0> closing AMQP connection <0.16537378.0> (<redacted>:40934 -> <redacted>:5671 - api_server_s4lSimcore1-2_778.02, vhost: '/', user: '<redacted>'):
Exceptions from our python client:
{"exception_type": "<class 'aiormq.exceptions.AMQPConnectionError'>", "exception_string": "Server connection reset: ConnectionResetError(104, 'Connection reset by peer')", "exception_causes": "[CancelledError]''", "error_code": null, "context": { "sender": "amqps://<redacted>:******@<redacted>.mq.us-east-1.amazonaws.com:5671/?name=api_server_s4lSimcore1-2_778.02&heartbeat=60" }
RabbitMQ channel closed with unexpected error. { "exception_type": "<class 'aiormq.exceptions.AMQPConnectionError'>", "exception_string": "Server connection reset: ConnectionResetError(104, 'Connection reset by peer')", "exception_causes": "[CancelledError]''", "error_code": null, "context": { "sender": "2" }, "tip": null }
Configuration
- Instance Type:
mq.m5.large - Public accessibility:
true - Engine:
RabbitMQ - Deployment mode:
Cluster deployment - Engine version:
3.13.7 - Amazon MQ Configuration
# Default RabbitMQ delivery acknowledgement timeout is 30 minutes consumer_timeout = 1800000
- Newest
- Most votes
- Most comments
The "Connection reset by peer" errors you're experiencing with your Amazon MQ for RabbitMQ cluster could be caused by several factors:
-
Network issues: Intermittent network connectivity problems between your clients and the RabbitMQ cluster could cause connection resets.
-
Memory pressure: If the RabbitMQ broker is experiencing high memory usage, it might trigger memory alarms that affect client connections. When RabbitMQ raises a high memory alarm (when RabbitMQMemUsed exceeds RabbitMQMemLimit), it can impact connections.
-
Disk space issues: Similar to memory alarms, disk alarms can be triggered when there's a high number of unconsumed messages while new messages are being added, potentially affecting connections.
-
Connection/channel limits: Your cluster might be hitting connection or channel limits. For an m5.large instance, there are limits on connections (5,000) and channels (15,000).
-
Client-side issues: Your client library might not be properly handling connection recovery.
To further debug the problem:
-
Monitor CloudWatch metrics: Regularly check memory usage (RabbitMQMemUsed and RabbitMQMemLimit) and disk usage (RabbitMQDiskFree and RabbitMQDiskFreeLimit) metrics to identify potential resource constraints.
-
Implement automatic network recovery: Ensure your client code properly implements automatic network recovery. While you're using aiopika, make sure it's configured to handle connection failures and recovery.
-
Implement retry logic: Add retry logic for initial connection failures in your client code.
-
Check queue synchronization: If you're experiencing issues during maintenance windows, it could be related to queue synchronization. Consider adjusting the ha-sync-batch-size if synchronization is pausing.
-
Review client heartbeat settings: You mentioned a heartbeat of 60 seconds in your connection string. Consider adjusting this value to detect connection issues earlier.
-
Examine connection patterns: Look for patterns in when these errors occur - are they happening during specific operations, at certain times of day, or during maintenance windows?
-
Check for broker restarts: AWS might be performing maintenance on the cluster nodes, which could temporarily affect connections.
Since you mentioned this doesn't happen in your self-managed RabbitMQ cluster, it might be related to how Amazon MQ manages the RabbitMQ service, particularly around maintenance windows or resource management.
Sources
Amazon MQ for RabbitMQ sizing guidelines - Amazon MQ
Troubleshooting Amazon MQ - Amazon MQ
Resolving RabbitMQ paused queue synchronization - Amazon MQ
Best practices for network resilience and monitoring in Amazon MQ for RabbitMQ - Amazon MQ
answered 10 months ago
Once in a while some containers would restart because RabbitMQ connection breaks Server connection reset: ConnectionResetError(104, 'Connection reset by peer') and our client does not recover (until container starts new).
Amazon MQ recommends always enabling automatic network recovery [1] to prevent significant downtime in cases where client connections to RabbitMQ nodes fail.
Disconnections can happen for a variety of normal and expected reasons, like during Amazon MQ broker maintenance window [2] restarts or during instance type [3] upgrades. Please update your client application code to retry connections on disconnect instead of only once on container startup.
[1] https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/best-practices-network-resilience.html [2] https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/maintaining-brokers.html [3] https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/upgrading-instance-type.html
answered 10 months ago
Relevant content
asked 2 years ago
asked 5 years ago
Brokers's metrics (CPU, Memory usage) looks the same and healthy