kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammed Manna <manme...@gmail.com>
Subject Re: Cluster in weird state: no leaders no ISR for all topics, but it works!
Date Thu, 01 Jun 2017 15:05:49 GMT
I had a similar situation, but only 1 of my ZKs was struggling - but since
the ISR synching time is configurable I was confident to bounce 1 ZK at a
time and it worked out.
does it happen even when you create a new topic with a
replication:partition ration of 1?

i meant, 3 replicas, 3 partitions :)

On 1 June 2017 at 15:58, Del Barrio, Alberto <
alberto.delbarrio@360dialog.com> wrote:

> Hi Mohammed,
>
> thanks for your answer.
> The ZK cluster is not located in the servers where Kafka runs but in other
> 3 different machines. This ZK cluster is used by several other services
> which are not reporting problems.
> As you suggested, I haven't tried restarting the kafka-server processes
> because there's no leader for topic partitions, so I don't know what will
> happen. Never been in a similar situation with Kafka after some years using
> it.
>
>
>
>
> On 1 June 2017 at 16:29, Mohammed Manna <manmedia@gmail.com> wrote:
>
> > Hi Alberto,
> >
> > Usually this means that the leader election/replica syncing couldn't be
> > successful and the zookeeper logs should be able to show this information
> > too. The leader -1 is what worries me. For your case (3 broker cluster),
> I
> > am assuming you have done the cluster configuration to have 1
> > broker-zookeeper setup ?
> > If that's the case, you should be able to bounce 1 zookeeper at a time
> and
> > see if that resolves the issue.
> >
> > That said, have you restarted  your servers since this issue surfaced?
> >
> > On 1 June 2017 at 14:11, Del Barrio, Alberto <
> > alberto.delbarrio@360dialog.com> wrote:
> >
> > > Hi all,
> > >
> > > I'm experiencing an issue which I don't know how to solve, so I'm
> trying
> > to
> > > find some guidance on the topic.
> > >
> > > I have a cluster composed by 3 servers, one broker per server running
> > Kafka
> > > 0.10.0.1-1 which runs in production with around 100 topics, most of
> them
> > > divided in several partitions and replicated always between 2 servers.
> > > Suddenly I've notice when looking at my topics (with kafka-topics tool)
> > > that no one of them have a leader (Leader: -1) and the list of ISR
> > appears
> > > empty for all the topics.
> > > So they look something like:
> > >
> > > Topic:mytopic   PartitionCount:3    ReplicationFactor:2    Configs:
> > > retention.ms=86400000
> > >     Topic: mytopic    Partition: 0    Leader: -1    Replicas: 30,10
> > Isr:
> > >     Topic: mytopic    Partition: 1    Leader: -1    Replicas: 10,20
> > Isr:
> > >     Topic: mytopic    Partition: 2    Leader: -1    Replicas: 20,30
> > Isr:
> > >
> > > However the applications using it are running normally, consumers as
> well
> > > as producers.
> > > The logs are not showing errors or weird messages with the exceptions
> of
> > > some
> > > Failed to rename [/var/log/kafka/log-cleaner.log] to
> > > [/var/log/kafka/log-cleaner.log.2017-05-31-17]
> > > which appear each few days.
> > >
> > > Now I would like to bring back to cluster to a good state. I'm afraid
> of
> > > restarting brokers because all of them are supposed to be leaders for
> > some
> > > partitions, so if I restart them and there's no leader I might
> experience
> > > data loss.
> > >
> > > Have you face any similar situation? Can someone give me a hint?
> > >
> > > Thanks in advance,
> > > Alberto.
> > >
> >
>
>
>
> --
> News, jobs, product releases, events.
> Follow 360dialog on LinkedIn <https://www.linkedin.com/company/360dialog>
>  and Twitter <https://twitter.com/360dialog>.
> Subscribe to our newsletter <http://www.360dialog.com/newsletter/>.
>
>
> *Alberto del Barrio*DevOps Engineer
>
> <http://www.360dialog.com?utm_campaign=email-signature&utm_
> content=-&utm_medium=email&utm_source=signature&utm_term=logo>
>
>
>
> *Contact 360dialog*www.360dialog.com
> <http://www.360dialog.com/?utm_campaign=email-signature&
> utm_content=-&utm_medium=email&utm_source=signature&utm_term=url>
> info@360dialog.com
> +49-(0)30-6098-5953-0
>
> 360dialog GmbH, Saarbr├╝cker Str. 36-38, 10405 Berlin, Germany
> Managing director: Roland Siebert
> Commercial register: Charlottenbug, HRB 144188 B
> VAT ID: DE815382679
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message