Should MSK Security patching increase brokers CPU usage when done?
We had a security patching event on our cluster on September 6th, and while getting the expected spike in (user) CPU usage during the patching and immediately after it, it have now stabilized on a significantly higher level as compared to before the patching.
Is this something that would be expected (in my mind, it should not)?
Details: MSK Cluster running 3.2.0 using 6 brokers of the type kafka.t3.small using IAM for authentication and authorization.
Graph of current CPU usage (colored) compared to last weeks CPU usage (grey).
- Newest
- Most votes
- Most comments
Hello there,
Increased CPU Utilisation after the Security patch is not expected. There could be other reasons as well which needs investigation. To troubleshoot this issue further, please reach out to us over a Support Case.
If the CPU Utilisation continues to stay high after the patching event, I would suggest to restart the brokers which may helps to solve this issue. However, if that sorts out soon after the patching you may ignore it as the brokers require more CPU to handle the other broker's load whenever a broker goes down during the rolling restart.
I'm writing this as a follow-up response for anyone else encountering this.
The CPU usage stayed high for more than 6 days after the patching event, and in the end we took the decision to vertically scale the brokers (away from t3.small) to m5.large, in order to not encounter CPU throttling (when running out of credits).
With data stored on the cluster measured in Megabytes, number of connections to any single broker (we had 6) not exceeding 10, network traffic of less than 5 b/sec and staying below the recommended number of partitions per broker, the increased CPU load on the brokers are hard to explain by anything we as a customer are able to control.
Given the light load over all on the cluster, I also find it hard to accept that Kafka itself would require a week to internally redistribute data and reach a stable state.
As this is clearly within a normal use case for MSK, it is hard for me to understand why AWS would not like to investigate it (or at least not inform me as a customer who have reported it).
Relevant content
- asked 5 years ago
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
I'll create a case for this, even though the cluster in question have been vertically scaled since in order to handle this.
As we only have a basic support plan I created a support ticket as "General question" -> "General Info and Getting Started, Using AWS & Services" and were told I could not get help with this.
I still think this looks like more of a general issue with either slow recovery of the cluster after rotating a broker (when using t3.small, regardless of the amount of data on the cluster) or the recommended maximum number of partitions per broker (of which the limit we were still below in this case).