kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Del Barrio, Alberto" <alberto.delbar...@360dialog.com>
Subject Re: Cluster in weird state: no leaders no ISR for all topics, but it works!
Date Thu, 01 Jun 2017 15:49:31 GMT
Here are the configs you were asking for:

Zookeeper:
tickTime=1000
initLimit=2000
syncLimit=1000
dataDir=/var/lib/zookeeper
clientPort=2181
server.3=10.0.0.3:2888:3888
server.2=10.0.0.2:2888:3888
server.1=10.0.0.1:2888:3888


Kafka broker (for one of them):
broker.id=10
listeners=PLAINTEXT://10.0.0.4:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/var/lib/kafka
num.partitions=2
num.recovery.threads.per.data.dir=1
zookeeper.connect=10.0.0.1:2181,10.0.0.2:2181,10.0.0.3:2181/kafka
zookeeper.connection.timeout.ms=6000

In general they're pretty much the default ones.
I can see in Zookeeper the kafka brokers connected to it and exchanging
data...

Thanks for your help and time.

On 1 June 2017 at 17:32, Mohammed Manna <manmedia@gmail.com> wrote:

> Could you please share your broker/zookeeper/topic configs ?
>
> On 1 June 2017 at 16:18, Del Barrio, Alberto <
> alberto.delbarrio@360dialog.com> wrote:
>
> > I tried creating the topic and results are very similar to the current
> > situation: there are not ISR and no leader for any of the partitions, but
> > now kafka-topics shows *Leader: none* when for all the other topics, it
> > shows *Leader: -1*
> >
> >
> > On 1 June 2017 at 17:05, Mohammed Manna <manmedia@gmail.com> wrote:
> >
> > > I had a similar situation, but only 1 of my ZKs was struggling - but
> > since
> > > the ISR synching time is configurable I was confident to bounce 1 ZK
> at a
> > > time and it worked out.
> > > does it happen even when you create a new topic with a
> > > replication:partition ration of 1?
> > >
> > > i meant, 3 replicas, 3 partitions :)
> > >
> > > On 1 June 2017 at 15:58, Del Barrio, Alberto <
> > > alberto.delbarrio@360dialog.com> wrote:
> > >
> > > > Hi Mohammed,
> > > >
> > > > thanks for your answer.
> > > > The ZK cluster is not located in the servers where Kafka runs but in
> > > other
> > > > 3 different machines. This ZK cluster is used by several other
> services
> > > > which are not reporting problems.
> > > > As you suggested, I haven't tried restarting the kafka-server
> processes
> > > > because there's no leader for topic partitions, so I don't know what
> > will
> > > > happen. Never been in a similar situation with Kafka after some years
> > > using
> > > > it.
> > > >
> > > >
> > > >
> > > >
> > > > On 1 June 2017 at 16:29, Mohammed Manna <manmedia@gmail.com> wrote:
> > > >
> > > > > Hi Alberto,
> > > > >
> > > > > Usually this means that the leader election/replica syncing
> couldn't
> > be
> > > > > successful and the zookeeper logs should be able to show this
> > > information
> > > > > too. The leader -1 is what worries me. For your case (3 broker
> > > cluster),
> > > > I
> > > > > am assuming you have done the cluster configuration to have 1
> > > > > broker-zookeeper setup ?
> > > > > If that's the case, you should be able to bounce 1 zookeeper at a
> > time
> > > > and
> > > > > see if that resolves the issue.
> > > > >
> > > > > That said, have you restarted  your servers since this issue
> > surfaced?
> > > > >
> > > > > On 1 June 2017 at 14:11, Del Barrio, Alberto <
> > > > > alberto.delbarrio@360dialog.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'm experiencing an issue which I don't know how to solve, so
I'm
> > > > trying
> > > > > to
> > > > > > find some guidance on the topic.
> > > > > >
> > > > > > I have a cluster composed by 3 servers, one broker per server
> > running
> > > > > Kafka
> > > > > > 0.10.0.1-1 which runs in production with around 100 topics,
most
> of
> > > > them
> > > > > > divided in several partitions and replicated always between
2
> > > servers.
> > > > > > Suddenly I've notice when looking at my topics (with kafka-topics
> > > tool)
> > > > > > that no one of them have a leader (Leader: -1) and the list
of
> ISR
> > > > > appears
> > > > > > empty for all the topics.
> > > > > > So they look something like:
> > > > > >
> > > > > > Topic:mytopic   PartitionCount:3    ReplicationFactor:2
> Configs:
> > > > > > retention.ms=86400000
> > > > > >     Topic: mytopic    Partition: 0    Leader: -1    Replicas:
> 30,10
> > > > > Isr:
> > > > > >     Topic: mytopic    Partition: 1    Leader: -1    Replicas:
> 10,20
> > > > > Isr:
> > > > > >     Topic: mytopic    Partition: 2    Leader: -1    Replicas:
> 20,30
> > > > > Isr:
> > > > > >
> > > > > > However the applications using it are running normally, consumers
> > as
> > > > well
> > > > > > as producers.
> > > > > > The logs are not showing errors or weird messages with the
> > exceptions
> > > > of
> > > > > > some
> > > > > > Failed to rename [/var/log/kafka/log-cleaner.log] to
> > > > > > [/var/log/kafka/log-cleaner.log.2017-05-31-17]
> > > > > > which appear each few days.
> > > > > >
> > > > > > Now I would like to bring back to cluster to a good state. I'm
> > afraid
> > > > of
> > > > > > restarting brokers because all of them are supposed to be leaders
> > for
> > > > > some
> > > > > > partitions, so if I restart them and there's no leader I might
> > > > experience
> > > > > > data loss.
> > > > > >
> > > > > > Have you face any similar situation? Can someone give me a hint?
> > > > > >
> > > > > > Thanks in advance,
> > > > > > Alberto.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > News, jobs, product releases, events.
> > > > Follow 360dialog on LinkedIn <https://www.linkedin.com/
> > company/360dialog
> > > >
> > > >  and Twitter <https://twitter.com/360dialog>.
> > > > Subscribe to our newsletter <http://www.360dialog.com/newsletter/>.
> > > >
> > > >
> > > > *Alberto del Barrio*DevOps Engineer
> > > >
> > > > <http://www.360dialog.com?utm_campaign=email-signature&utm_
> > > > content=-&utm_medium=email&utm_source=signature&utm_term=logo>
> > > >
> > > >
> > > >
> > > > *Contact 360dialog*www.360dialog.com
> > > > <http://www.360dialog.com/?utm_campaign=email-signature&
> > > > utm_content=-&utm_medium=email&utm_source=signature&utm_term=url>
> > > > info@360dialog.com
> > > > +49-(0)30-6098-5953-0
> > > >
> > > > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany
> > > > Managing director: Roland Siebert
> > > > Commercial register: Charlottenbug, HRB 144188 B
> > > > VAT ID: DE815382679
> > > >
> > >
> >
> >
> >
> > --
> > News, jobs, product releases, events.
> > Follow 360dialog on LinkedIn <https://www.linkedin.com/company/360dialog
> >
> >  and Twitter <https://twitter.com/360dialog>.
> > Subscribe to our newsletter <http://www.360dialog.com/newsletter/>.
> >
> >
> > *Alberto del Barrio*DevOps Engineer
> >
> > <http://www.360dialog.com?utm_campaign=email-signature&utm_
> > content=-&utm_medium=email&utm_source=signature&utm_term=logo>
> >
> >
> >
> > *Contact 360dialog*www.360dialog.com
> > <http://www.360dialog.com/?utm_campaign=email-signature&
> > utm_content=-&utm_medium=email&utm_source=signature&utm_term=url>
> > info@360dialog.com
> > +49-(0)30-6098-5953-0
> >
> > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany
> > Managing director: Roland Siebert
> > Commercial register: Charlottenbug, HRB 144188 B
> > VAT ID: DE815382679
> >
>



-- 
News, jobs, product releases, events.
Follow 360dialog on LinkedIn <https://www.linkedin.com/company/360dialog>
 and Twitter <https://twitter.com/360dialog>.
Subscribe to our newsletter <http://www.360dialog.com/newsletter/>.


*Alberto del Barrio*DevOps Engineer

<http://www.360dialog.com?utm_campaign=email-signature&utm_content=-&utm_medium=email&utm_source=signature&utm_term=logo>



*Contact 360dialog*www.360dialog.com
<http://www.360dialog.com/?utm_campaign=email-signature&utm_content=-&utm_medium=email&utm_source=signature&utm_term=url>
info@360dialog.com
+49-(0)30-6098-5953-0

360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany
Managing director: Roland Siebert
Commercial register: Charlottenbug, HRB 144188 B
VAT ID: DE815382679

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message