kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Del Barrio, Alberto" <alberto.delbar...@360dialog.com>
Subject Cluster in weird state: no leaders no ISR for all topics, but it works!
Date Thu, 01 Jun 2017 13:11:24 GMT
Hi all,

I'm experiencing an issue which I don't know how to solve, so I'm trying to
find some guidance on the topic.

I have a cluster composed by 3 servers, one broker per server running Kafka
0.10.0.1-1 which runs in production with around 100 topics, most of them
divided in several partitions and replicated always between 2 servers.
Suddenly I've notice when looking at my topics (with kafka-topics tool)
that no one of them have a leader (Leader: -1) and the list of ISR appears
empty for all the topics.
So they look something like:

Topic:mytopic   PartitionCount:3    ReplicationFactor:2    Configs:
retention.ms=86400000
    Topic: mytopic    Partition: 0    Leader: -1    Replicas: 30,10    Isr:
    Topic: mytopic    Partition: 1    Leader: -1    Replicas: 10,20    Isr:
    Topic: mytopic    Partition: 2    Leader: -1    Replicas: 20,30    Isr:

However the applications using it are running normally, consumers as well
as producers.
The logs are not showing errors or weird messages with the exceptions of
some
Failed to rename [/var/log/kafka/log-cleaner.log] to
[/var/log/kafka/log-cleaner.log.2017-05-31-17]
which appear each few days.

Now I would like to bring back to cluster to a good state. I'm afraid of
restarting brokers because all of them are supposed to be leaders for some
partitions, so if I restart them and there's no leader I might experience
data loss.

Have you face any similar situation? Can someone give me a hint?

Thanks in advance,
Alberto.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message