How do I troubleshoot “Persistent store is Full” errors in Amazon MQ for ActiveMQ?

Short description

Amazon MQ for ActiveMQ returns a Persistent store is Full error to CloudWatch when a broker reaches its storage limit. When an Amazon MQ for ActiveMQ broker reaches its storage limit, the broker can't receive messages.

To resolve Persistent store is Full errors, first confirm the cause of the error by reviewing the following CloudWatch metrics for your broker:

• Store Percentage Usage
• Journal Files for Full Recovery
• Journal Files for Fast Recovery

Then, based on your broker's metrics, reduce the amount of data that's saved to the broker's persistent storage as required.

To help prevent the error from occurring again, see the Best practices for preventing "Persistent store is Full" errors section of this article. For more information, see AMQ Message Store and Producer Flow Control in the Apache ActiveMQ documentation.

Example CloudWatch "Persistent store is Full" error message for an Amazon MQ for ActiveMQ broker

INFO | Usage(default:store:queue://broker.name:store) percentUsage=99%, usage=537210471, limit=536870912, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%, usage=537210471, limit=536870912,percentUsageMinDelta=1%: Persistent store is Full, 100% of 536870912. Stopping producer (ID:xxxx) to prevent flooding queue://broker.name. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 155s)

Resolution

Confirm the cause of the error

1.    Open the CloudWatch console.

Important: Make sure that you log in to the same AWS Region that your broker is in.

2.    In the left navigation pane, choose Metrics. Then, choose All metrics. The CloudWatch Metrics page opens.

3.    On the Metrics page, choose Amazon MQ. Then, choose Broker Metrics.

4.    In the search box, enter your broker's name. Then, press Enter on your keyboard. All of the CloudWatch metrics for the broker that you entered in the search box appear.

5.    Review the Store Percentage Usage metric value. If your broker's Store Percentage Usage metric reaches 100, the broker stops receiving messages and returns a Persistent store is Full error.

6.    If your broker's Store Percentage Usage metric is 100 (or close to 100), review the following metrics' values to determine the cause:

• **Journal Files for Full Recovery—**indicates the number of journal files that are replayed after a clean shutdown. An increase in this metric indicates that either your broker's consumers are running slow, or that there are too many pending messages in persistent storage.
• **Journal Files for Fast Recovery—**indicates the number of journal files that are replayed after an unclean shutdown. An increase in this metric indicates that there are too many pending messages in persistent storage.

Note: An ActiveMQ KahaDB journal log file (data file) can remain in persistent storage for any of the following reasons:

• The data file includes a pending message for a durable topic subscription or destination.
• The data file includes an individual Acknowledgement (ACK) for a message that's in an in-use data file.
• The data file references a pending transaction.
• The data file is a journal file, and there's a pending write action to it.

For more information, see Why do KahaDB log files remain after cleanup in the ActiveMQ documentation.

To reduce the amount of data that's saved to an Amazon MQ for ActiveMQ broker's persistent storage

Consume or delete the pending messages that are in the broker's queues and durable topics

For instructions, see How do I purge a queue in the Apache ActiveMQ documentation.

Note: Amazon MQ for ActiveMQ doesn't support Java Management Extensions (JMX) currently. You must use the ActiveMQ Web Console to purge your broker's queues and durable topics.

Consume or delete expired messages that are in the broker's dead-letter queue (DLQ)

For instructions, see the Automatically discard expired messages section of Message redelivery and DLQ handling in the Apache ActiveMQ documentation.

Note: To consume messages in your broker's DLQ, first configure the DLQ as a regular queue. Then, configure your consumers to poll the DLQ directly.

Best practices for preventing "Persistent store is Full" errors

Verify that you've chosen the correct broker instance type for your use case

For more information, see Choose the correct broker instance type for the best throughput in the Amazon MQ Developer Guide.

Note: Depending on your use case, adding consumers to your broker can sometimes help the broker's message throughput.

Increase the number and speed of consumers that you're using

When you use ActiveMQ in persistent mode, writing to storage usually occurs when there are either too few consumers or when the consumers are slow. Increasing the number and speed of consumers that you're using can help reduce the amount of data your broker writes to persistent storage.

Set a CloudWatch alarm based on your broker's "Store Percentage Usage" metric

If your broker's Store Percentage Usage metric reaches 100 (or close to 100), the broker stops receiving messages. To have CloudWatch alert you when this metric reaches a specific threshold, you can set an alarm in CloudWatch. Then, you can take the preventative actions described in this article before the error occurs.

For instructions, see Create a CloudWatch alarm based on a static threshold in the CloudWatch User Guide.

---

Related information

Ensuring effective Amazon MQ performance

Logging and monitoring Amazon MQ for ActiveMQ brokers