kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudia Wegmann <c.wegm...@kasasi.de>
Subject Instable Kafka Cluster
Date Thu, 07 Mar 2019 14:40:04 GMT
Dear kafka experts,

I've got a kafka cluster with 3 brokers running in docker-containers on different hosts in
version 2.1.1 of kafka. The cluster is serving some kafka streams apps. The topics are configured
with replication.factor 3 and min.insync.replicas 2.  The cluster works fine most of the time.

But just now, I encountered strange behavior. One broker (id 1) was suddenly out of the blue
starting to log ISR Shrinks for  for some partitions (not all)  it was leading:
"INFO [Partition __transaction_state-42 broker=1] Shrinking ISR from 1,3,2 to 1 (kafka.cluster.Partition)"

About 2 seconds later it started to expand these ISR again while some other partitions where
still shrinking.
Three minutes later, this scenario repeated itself. 

I could not find anything other that was out of the ordinary neither in the logs of the kafka
brokers nor in the metrics (network, CPU, RAM, disk,...).
This leads to the question why the two brokers fell out of the ISR or the other broker at
least had the impression the other two did? Has anyone experienced similar behavior? Or is
this expected an can happen from time to time?

I will definitely set up retries and retry.backoff.ms config the avoid the streams applications
going down because of a small hunch like this in the future.

Thanks for your input.
View raw message