Why can't I increase the Amazon EBS disk volume in my Amazon MSK cluster?

Resolution

The most common causes for this issue are as follows:

• You don't have the required permissions to modify the Amazon EBS volume.
• Your cluster is in the optimizing state from a prior Amazon EBS volume upgrade.

You don't have the required permissions to modify the Amazon EBS volume

Make sure that you have the required permissions to modify the EBS disk volume for your cluster. Make sure that you attached the AmazonMSKFullAccess or AmazonMSKClusterManagement policy to your AWS Identity and Access Management (IAM) user or role.

Your cluster is in the optimizing state from a prior Amazon EBS volume upgrade

In this case, you get the error "An EBS volume in this cluster is under modification from a previous operation".

When you increase the size of an EBS volume that's attached to your Amazon MSK broker node, the newly added storage is available to you immediately. However, it takes from 6 to 24 hours or more for the cluster to be optimized. The duration for cluster optimization is proportional to the size of the storage.

When you increase the size of the EBS volume, make sure that the added volume is sufficient for the next 48 hours or more. During this period, you can't add more volumes to your cluster. You can't update the volume further until the cluster is optimized.

Important: Suppose that your cluster is being optimized and you're already running out of disk space. In this case, be sure to turn off automatic scaling if you previously turned it on. Otherwise, when optimization is complete, the volume is increased only by 10% or 10 GB. This might be too low, causing similar issues with volume modification. After turning off automatic scaling, be sure to analyze your traffic patterns and increase your storage space accordingly.

If your cluster is undergoing modification and the KafkaDataLogsDiskUsed metric is less than 94%, you can delete unused or old topics to free up space. Run the following command to determine the space occupied per topic:

kafka-topics.sh --command-config /tmp/client.properties --bootstrap-server $bservers --list | xargs -I{} sh -c "echo -n '{} - ' && kafka-log-dirs.sh --command-config /tmp/client.properties --bootstrap-server $bservers --topic-list {} --describe | grep '^{'  | jq '[ ..|.size? | numbers ] | add' | numfmt --to iec " | tee /tmp/topics-by-size.list

Then, delete the unused or old topics to free up disk space.

Note: If KafkaDataLogsDiskUsed is higher than 94%, then this command might be unresponsive. In this case, you can't delete the topics.