How is it possible to remove a specific broker from an MSK cluster (due to 100% CPU)?

Recently we went through one of the worst incidents i have been a part of. Much of our infrastructure is supported by Kafka for the various event messages that the different applications create, among other uses. What happened is that one of the brokers managed to get itself corrupted, or more accurately in a very bad state, where its CPU went to 100% (normal CPU, ~15%), making it very challenging for producers. This node was also a leader on several partitions in each topic, although there was sufficient RF (2, for each topic, across 4 brokers) where if a node went down there would not be data loss. Producers would either take a very, very long time to produce a message (> 1000ms), or worse still, the operation would fail. Although the applications were configured to timeout after a certain amount of time, or, to swallow the errors, to avoid service interruption should the kafka operation fail, many of our processes depend on that data. And frankly, this situation wasn't really something we were well prepared for. all of the discussions had always been around "what if a node fails", and not, "What if a node is operating so poorly, it affects all applications connected to this cluster". Restarting the node was an option, and... didn't really help. After restart it seemed more improved, back down to normal, but didn't take that long before it hit 100% cpu again and causing issues with producers all through the environment.

On the MSK screen, there is the choice to remove nodes, but this seems to be specifically about downsizing the cluster. we do not get to choose the nodes we want removed, and we must remove the same number of nodes in each AZ. so this option seems unfit for this purpose. Restarting is an option, but not the one that we want. If this had been my bare metal kafka cluster we would have just removed the offending broker, probably add a replacement and be done with it.

What we ended up doing was altering the machine size of the brokers, forcing each broker to be rebuilt, which ultimately solved the issue with our offending broker. I am hoping there would be a better way in which to handle this. It's not the end of the world if that is our only option, or, whether we would be better off with a self-managed cluster. but now we know it is entirely possible to suffer from the poison pill effect, i am keen to know what the experts would recommend should such a thing or similar happen again. one hopes the likelihood of a repeat would be low (not seen this behavior in 15 years of working with kafka), but we have to be able to recover gracefully and in a timely fashion, which make the broker resize methodology slightly less attractive (takes about 1 hour, and has caused some challenges with consumer offsets getting corrupted and issues downstream). suggestions / advice welcome.

Furthermore, documentation suggests that after some time our cluster would go into a healing state, or that the managed service aspect would attempt to repair our not-very-responsive broker. This did not happen. It was in this state of high CPU for many hours (> 12). it took some time to actually see this was happening, through too many failed attempts to produce on the upstream clients, then we had the whole debacle around restarting, or adding more brokers to the cluster and trying to reassign partitions away from the poisoned broker. Incidentally it was impossible to reassign all of the partitions because the broker in question was unresponsive; the operation would simply hang and wait on those partitions where the damaged broker was the assigned leader.

Edit: using v.2.8.1 for anyone interested. the AI answer seems to suggest that we can remove the specific broker via the console, or via the update_broker_count, but, and possibly it is my lack of basic comprehension skills, i don't see it asking for the broker-id, or arn of the offending broker, so a little bit puzzled on how that might happen. the worst possible thing to happen would be to remove healthy broker(s), leaving our poisoned broker responsible for a greater % of requests / partitions.