Can not connect to kafka brokers but zookeepers are still

Question

My msk service are running well until recently kafka brokers cannot be connected but zookeepers are still connected successfully.  
  
I also double checked VPC, It's same. Eventually restart all brokers  
  
I tried these commands below  
```  
telnet broker_dns_1 9092  
#=> Trying ip...  
telnet: Unable to connect to remote host: Connection refused  
  
telnet zookeeper_dns_1 2181  
#=> Trying ip...  
connected to zookeeper_dns  
```  
  
Because response of telnet broker_dns command are so fast, I guess they can be blocked by some rules. have you guys encountered this issue or known the root cause ?  
  
Edited by: taisytran on Jan 17, 2021 8:50 PM  
  
Edited by: taisytran on Jan 17, 2021 8:50 PM

Answer

I've resolved, out of storage problem

Answer

I double checked broker status on my monitoring tool.  
  
Broker state is 2: Recovering from Unclean Shutdown extends Broker States (https://docs.itrsgroup.com/docs/geneos/5.3.0/Integrations/Kafka/kafka_monitoring_tr.html)  
  
follow this page, I guess my brokers are stuck on this state

Can not connect to kafka brokers but zookeepers are still

Relevant content