How do I troubleshoot “Persistent store is Full” errors in Amazon MQ for ActiveMQ?
5 minute read
I can't send messages to my Amazon MQ for ActiveMQ broker. When I review the broker's logs in Amazon CloudWatch, I see a "Persistent store is Full" error. What's causing the error, and how do I troubleshoot the issue?
Amazon MQ for ActiveMQ returns a Persistent store is Full error to CloudWatch when a broker reaches its storage limit. When an Amazon MQ for ActiveMQ broker reaches its storage limit, the broker can't receive messages.
To resolve Persistent store is Full errors, first confirm the cause of the error by reviewing the following CloudWatch metrics for your broker:
Store Percentage Usage
Journal Files for Full Recovery
Journal Files for Fast Recovery
Then, based on your broker's metrics, reduce the amount of data that's saved to the broker's persistent storage as required.
To help prevent the error from occurring again, see the Best practices for preventing "Persistent store is Full" errors section of this article. For more information, see AMQ Message Store and Producer Flow Control in the Apache ActiveMQ documentation.
Example CloudWatch "Persistent store is Full" error message for an Amazon MQ for ActiveMQ broker
INFO | Usage(default:store:queue://broker.name:store) percentUsage=99%, usage=537210471, limit=536870912, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%, usage=537210471, limit=536870912,percentUsageMinDelta=1%: Persistent store is Full, 100% of 536870912. Stopping producer (ID:xxxx) to prevent flooding queue://broker.name. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 155s)
Important: Make sure that you log in to the same AWS Region that your broker is in.
2. In the left navigation pane, choose Metrics. Then, choose All metrics. The CloudWatch Metrics page opens.
3. On the Metrics page, choose Amazon MQ. Then, choose Broker Metrics.
4. In the search box, enter your broker's name. Then, press Enter on your keyboard. All of the CloudWatch metrics for the broker that you entered in the search box appear.
5. Review the Store Percentage Usage metric value. If your broker's Store Percentage Usage metric reaches 100, the broker stops receiving messages and returns a Persistent store is Full error.
6. If your broker's Store Percentage Usage metric is 100 (or close to 100), review the following metrics' values to determine the cause:
**Journal Files for Full Recovery—**indicates the number of journal files that are replayed after a clean shutdown. An increase in this metric indicates that either your broker's consumers are running slow, or that there are too many pending messages in persistent storage.
**Journal Files for Fast Recovery—**indicates the number of journal files that are replayed after an unclean shutdown. An increase in this metric indicates that there are too many pending messages in persistent storage.
Note: An ActiveMQ KahaDB journal log file (data file) can remain in persistent storage for any of the following reasons:
The data file includes a pending message for a durable topic subscription or destination.
The data file includes an individual Acknowledgement (ACK) for a message that's in an in-use data file.
The data file references a pending transaction.
The data file is a journal file, and there's a pending write action to it.
Note: Depending on your use case, adding consumers to your broker can sometimes help the broker's message throughput.
Increase the number and speed of consumers that you're using
When you use ActiveMQ in persistent mode, writing to storage usually occurs when there are either too few consumers or when the consumers are slow. Increasing the number and speed of consumers that you're using can help reduce the amount of data your broker writes to persistent storage.
Set a CloudWatch alarm based on your broker's "Store Percentage Usage" metric
If your broker's Store Percentage Usage metric reaches 100 (or close to 100), the broker stops receiving messages. To have CloudWatch alert you when this metric reaches a specific threshold, you can set an alarm in CloudWatch. Then, you can take the preventative actions described in this article before the error occurs.