samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Question on hello-samza (Kafka startup and shutdown)
Date Thu, 19 Feb 2015 22:01:48 GMT
Depending on the version of Kafka you're at, "controlled.shutdown.enable"
should be set to true. If that's true and you always shutdown the broker
cleanly (kill -15, not kill -9) and there are more than 1 replicas
available, you should not see LeaderNotAvailable exceptions. If you kill
the broker (kill -9) then Kafka does not get a chance to move the leaders
away from the broken being shut down and the leader re-election can take
some time leading to many LeaderNotAvailable exceptions.

You can verify the replica availability as well as leader movement through
the kafka-topics command before shutting down zookeeper.

Thanks
Neha

On Thu, Feb 19, 2015 at 10:51 AM, Felix GV <fvillegas@linkedin.com.invalid>
wrote:

> I'm not 100% sure, but I think this happens when ZK ephemeral znodes have
> not had time to expire properly. When Kafka shuts down gracefully, it
> should clean up its ephemeral nodes immediately (presumably, but that is
> also an assumption... maybe it does have a short-coming in its graceful
> shutdown logic). If Kafka gets killed improperly and bounced back up right
> away, it cannot assume leadership properly because the ephemeral znodes of
> the previous run are still there in ZK.
>
> I imagine Kafka could have some logic to deal with that better when it
> gets fast-bounced... Alternatively, you may just have to wait a bit before
> restarting Kafka after killing it.
>
> If anyone knows better, please correct me if I'm wrong.
>
> --
>
> Felix GV
> Data Infrastructure Engineer
> Distributed Data Systems
> LinkedIn
>
> fgv@linkedin.com
> linkedin.com/in/felixgv
>
> ________________________________________
> From: Chinmay Soman [chinmay.cerebro@gmail.com]
> Sent: Thursday, February 19, 2015 10:44 AM
> To: dev@samza.apache.org
> Subject: Question on hello-samza (Kafka startup and shutdown)
>
> Sending to a wider audience to know if anyone is also seeing this issue.
>
> It seems Kafka gets in a weird state everytime I do bin/grid stop all  (and
> then start all).
>
> I keep getting a LeaderNotAvailable exception on the producer side. It
> seems this happens everytime Kafka hasn't been shut down properly. This
> issue goes away if I use the following sequence:
>
> * bin/grid stop kafka
> * bin/grid stop zookeeper (after like 5 seconds).
>
> (and then start everything).
>
> Has anyone else seen this ?
>
> --
> Thanks and regards
>
> Chinmay Soman
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message