Amazon MQ - RabbitMQ - RABBITMQ_DISK_ALARM - Node not running

When a RabbitMQ on Amazon MQ broker raises a disk limit alarm (RABBITMQ_DISK_ALARM), it indicates that the free disk space on a node has decreased below the disk free limit due to a high number of messages not being consumed while new messages were added.

In your situation, even though you've processed all visible messages, there are a few important points to understand:

1. The disk alarm is cluster-wide - if one node goes under the limit, all other nodes will block incoming messages.

2. Amazon MQ will not restart a broker experiencing a disk alarm and will return an exception for RebootBroker API operations as long as the broker continues to raise the alarm.

3. The "Node not running" status for one of your nodes could be related to the disk alarm issue, and there might indeed be messages stored on that node that aren't visible from the running nodes.

To resolve this issue:

• Create a new connection specifically to consume any remaining messages
• Consider purging messages from queues that might still have content
• You might need to delete problematic queues if they can't be cleared otherwise

Even after taking these actions, it may take several hours for the alarm status to clear. The system needs time to recognize that the disk usage has decreased sufficiently.

If the issue persists after 24+ hours, you might want to check the CloudWatch metrics, particularly MessageCount and RabbitMQDiskFree, to see if there's still a disk space issue. The metrics can help diagnose whether there are still messages or other data consuming disk space on your broker.
Sources
RabbitMQ on Amazon MQ: Disk limit alarm - Amazon MQ
Amazon MQ for RabbitMQ: Instance type change alarm - Amazon MQ

Amazon MQ will not restart a broker that's experiencing a disk alarm, and it will return an exception for RebootBroker API operations as long as the alarm persists. The RABBITMQ_DISK_ALARM indicates that disk usage has exceeded the configured threshold, which is a protective measure to prevent data loss or corruption.

Even though you've processed all visible messages, the alarm may not be clearing for a few reasons:

• The non-running node might still contain messages or data that aren't visible from the other nodes
• Disk space may not have been fully freed even after processing messages, particularly if you're using persistent messages
• The alarm threshold check may need additional time to detect the freed space

Given that you've already cleared the queues and waited over 24 hours without resolution, I'd recommend the following:

1. Open an AWS Support case with high priority - Since this is preventing normal operations and standard troubleshooting hasn't resolved it, AWS Support can investigate the non-running node's state and potentially clear the disk alarm manually if appropriate.

2. Check CloudWatch metrics - If you have access, review the RabbitMQDiskFree metric for all three nodes to confirm actual disk usage levels.

3. Be aware of potential hidden messages - Once the node is recovered, messages that were on the non-running node may reappear in your queues.

When Opening Your Support Case

Include the following information to help expedite resolution:

• Your broker ID and region
• Timeline of when the issue started
• Confirmation that all visible queues have been cleared
• Any CloudWatch metrics showing disk usage (if available)