Consumer is not rebalancing when AWS MSK maintenance going on

Question

We are observing that on the consumer side whenever consumer `max.poll.interval.ms` is breached while AWS Maintenance is going on then the consumer stops consuming messages from the topic. Basically, we see a log like below on the consumer side and then the consumer stops consuming messages.

```
[Consumer clientId=consumer-aws.fct.entityupdate.webhooklistener.consumer-14, groupId=aws.fct.entityupdate.webhooklistener.consumer] consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
```

We have two instances of consumer running and once `max.poll.interval.ms` gets breached on both (while AWS MSK maintenance is on) then the consumer group stops consuming message from Kafka. The only fix is to restart the application containing the consumers.

The problem happens only during AWS MSK Maintainance. If there are no maintainance  then it is working as expected and we are observing that once a poll timeout happens, the consumer leaves the group and new consumer joins the group and new consumer keeps on consuming messages.

* Amazon MSK with Kafka 2.8.1 (Provisoned)
* Apache Kafka Client 3.5.1
* 2 instances of Application running on ECS Fargate where one consumer instance running in both . Consumers belong to the same consumer group and consuming from same topic having 10 partitions.

Can someone let me know if this is expected during AWS MSK Maintaince ?

I do not see much logs on the broker side also.

Answer

Hello,

I would like to inform you that MSK performs Broker update and Patching workflows using a rolling restart process which allows MSK to remain available and durable that means no downtime is required from MSK end. However, it can affect your producers/consumers because broker may be unavailable for that particular time and the partition leadership moves from one broker to another as brokers are restarted.

Hence it is not expected to see the below error which you are observing during MSK maintenance since in a rolling restart other brokers are available while one goes offline. With that being said, this error seems to be because of poll latency which is an expected behavior during security patching as client will get connection timeout error.

(+) [Consumer clientId=consumer-aws.fct.entityupdate.webhooklistener.consumer-14, groupId=aws.fct.entityupdate.webhooklistener.consumer] consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.

Having said that, I would recommend you to increase max.poll.interval.ms value since this latency is impacting max.poll.interval.ms value . Please note that these settings need to be updated at the consumer side and are not configurable at cluster level. You need to update the client.properties to update these parameters.

For a deeper analysis into the issue and gain more insights tailored to your cluster and client configurations, I request you to please reach out to AWS Premium Support team via a support case.

Please find the documentation for best practices : https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html#ensure-high-availability

I hope the above information is helpful.

Consumer is not rebalancing when AWS MSK maintenance going on

Relevant content