samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Question on hello-samza (Kafka startup and shutdown)
Date Fri, 20 Feb 2015 16:47:24 GMT
Hey Chinmay,

It seems controlled.shutdown.enable=true is the default. Chinmay, did you
figure this out? I haven't seen this before, but I don't usually stop/start
within 5s of eachother.

One thing that you might have a look at is whether the Kafka or ZK
processes are living past bin/grid stop all. I have seen procs (NM and
Kafka usually) continue to be alive after `stop all` is executed. I think
this is because the stop scripts SIGTERM and return immediately. This
allows procs to do a cleaner shutdown. But if you stop/start quickly, you
might get some weirdness there. Try jps'ing in between the stop/start, and
check to make sure there's nothing still alive (wait in a loop until
everything shuts down cleanly, and kill -9 if it takes more than 60s, or
something).

Cheers,
Chris

On Thu, Feb 19, 2015 at 2:01 PM, Neha Narkhede <neha.narkhede@gmail.com>
wrote:

> Depending on the version of Kafka you're at, "controlled.shutdown.enable"
> should be set to true. If that's true and you always shutdown the broker
> cleanly (kill -15, not kill -9) and there are more than 1 replicas
> available, you should not see LeaderNotAvailable exceptions. If you kill
> the broker (kill -9) then Kafka does not get a chance to move the leaders
> away from the broken being shut down and the leader re-election can take
> some time leading to many LeaderNotAvailable exceptions.
>
> You can verify the replica availability as well as leader movement through
> the kafka-topics command before shutting down zookeeper.
>
> Thanks
> Neha
>
> On Thu, Feb 19, 2015 at 10:51 AM, Felix GV <fvillegas@linkedin.com.invalid
> >
> wrote:
>
> > I'm not 100% sure, but I think this happens when ZK ephemeral znodes have
> > not had time to expire properly. When Kafka shuts down gracefully, it
> > should clean up its ephemeral nodes immediately (presumably, but that is
> > also an assumption... maybe it does have a short-coming in its graceful
> > shutdown logic). If Kafka gets killed improperly and bounced back up
> right
> > away, it cannot assume leadership properly because the ephemeral znodes
> of
> > the previous run are still there in ZK.
> >
> > I imagine Kafka could have some logic to deal with that better when it
> > gets fast-bounced... Alternatively, you may just have to wait a bit
> before
> > restarting Kafka after killing it.
> >
> > If anyone knows better, please correct me if I'm wrong.
> >
> > --
> >
> > Felix GV
> > Data Infrastructure Engineer
> > Distributed Data Systems
> > LinkedIn
> >
> > fgv@linkedin.com
> > linkedin.com/in/felixgv
> >
> > ________________________________________
> > From: Chinmay Soman [chinmay.cerebro@gmail.com]
> > Sent: Thursday, February 19, 2015 10:44 AM
> > To: dev@samza.apache.org
> > Subject: Question on hello-samza (Kafka startup and shutdown)
> >
> > Sending to a wider audience to know if anyone is also seeing this issue.
> >
> > It seems Kafka gets in a weird state everytime I do bin/grid stop all
> (and
> > then start all).
> >
> > I keep getting a LeaderNotAvailable exception on the producer side. It
> > seems this happens everytime Kafka hasn't been shut down properly. This
> > issue goes away if I use the following sequence:
> >
> > * bin/grid stop kafka
> > * bin/grid stop zookeeper (after like 5 seconds).
> >
> > (and then start everything).
> >
> > Has anyone else seen this ?
> >
> > --
> > Thanks and regards
> >
> > Chinmay Soman
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message