kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammed Manna <manme...@gmail.com>
Subject Re: Cluster in weird state: no leaders no ISR for all topics, but it works!
Date Thu, 01 Jun 2017 14:29:37 GMT
Hi Alberto,

Usually this means that the leader election/replica syncing couldn't be
successful and the zookeeper logs should be able to show this information
too. The leader -1 is what worries me. For your case (3 broker cluster), I
am assuming you have done the cluster configuration to have 1
broker-zookeeper setup ?
If that's the case, you should be able to bounce 1 zookeeper at a time and
see if that resolves the issue.

That said, have you restarted  your servers since this issue surfaced?

On 1 June 2017 at 14:11, Del Barrio, Alberto <
alberto.delbarrio@360dialog.com> wrote:

> Hi all,
>
> I'm experiencing an issue which I don't know how to solve, so I'm trying to
> find some guidance on the topic.
>
> I have a cluster composed by 3 servers, one broker per server running Kafka
> 0.10.0.1-1 which runs in production with around 100 topics, most of them
> divided in several partitions and replicated always between 2 servers.
> Suddenly I've notice when looking at my topics (with kafka-topics tool)
> that no one of them have a leader (Leader: -1) and the list of ISR appears
> empty for all the topics.
> So they look something like:
>
> Topic:mytopic   PartitionCount:3    ReplicationFactor:2    Configs:
> retention.ms=86400000
>     Topic: mytopic    Partition: 0    Leader: -1    Replicas: 30,10    Isr:
>     Topic: mytopic    Partition: 1    Leader: -1    Replicas: 10,20    Isr:
>     Topic: mytopic    Partition: 2    Leader: -1    Replicas: 20,30    Isr:
>
> However the applications using it are running normally, consumers as well
> as producers.
> The logs are not showing errors or weird messages with the exceptions of
> some
> Failed to rename [/var/log/kafka/log-cleaner.log] to
> [/var/log/kafka/log-cleaner.log.2017-05-31-17]
> which appear each few days.
>
> Now I would like to bring back to cluster to a good state. I'm afraid of
> restarting brokers because all of them are supposed to be leaders for some
> partitions, so if I restart them and there's no leader I might experience
> data loss.
>
> Have you face any similar situation? Can someone give me a hint?
>
> Thanks in advance,
> Alberto.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message