kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clark Breyman <cl...@breyman.com>
Subject Re: Controlled shutdown and leader election issues
Date Wed, 02 Apr 2014 19:59:34 GMT
Was there an answer for 0.8.1 getting stuck in preferred leader election?
I'm seeing this as well. Is there a JIRA ticket on this issue?


On Fri, Mar 21, 2014 at 1:15 PM, Ryan Berdeen <rberdeen@hubspot.com> wrote:

> So, for 0.8 without "controlled.shutdown.enable", why would ShutdownBroker
> and restarting cause under-replication and producer exceptions? How can I
> upgrade gracefully?
>
> What's up with 0.8.1 getting stuck in preferred leader election?
>
>
> On Fri, Mar 21, 2014 at 12:18 AM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Which brings up the question - Do we need ShutdownBroker anymore? It
> seems
> > like the config should handle controlled shutdown correctly anyway.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Mar 20, 2014 at 9:16 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> > > We haven't been testing the ShutdownBroker command in 0.8.1 rigorously
> > > since in 0.8.1, one can do the controlled shutdown through the new
> config
> > > "controlled.shutdown.enable". Instead of running the ShutdownBroker
> > command
> > > during the upgrade, you can also wait until under replicated partition
> > > count drops to 0 after each restart before moving to the next one.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Mar 20, 2014 at 3:14 PM, Ryan Berdeen <rberdeen@hubspot.com>
> > > wrote:
> > >
> > > > While upgrading from 0.8.0 to 0.8.1 in place, I observed some
> > surprising
> > > > behavior using kafka.admin.ShutdownBroker. At the start, there were
> no
> > > > underreplicated partitions. After running
> > > >
> > > >   bin/kafka-run-class.sh kafka.admin.ShutdownBroker --broker 10 ...
> > > >
> > > > Partitions that had replicas on broker 10 were under-replicated:
> > > >
> > > >   bin/kafka-topics.sh --describe --under-replicated-partitions ...
> > > >   Topic: analytics-activity Partition: 2  Leader: 12  Replicas: 12,10
> > > Isr:
> > > > 12
> > > >   Topic: analytics-activity Partition: 6  Leader: 11  Replicas: 11,10
> > > Isr:
> > > > 11
> > > >   Topic: analytics-activity Partition: 14 Leader: 14  Replicas: 14,10
> > > Isr:
> > > > 14
> > > >   ...
> > > >
> > > > While restarting the broker process, many produce requests failed
> with
> > > > kafka.common.UnknownTopicOrPartitionException.
> > > >
> > > > After each broker restart, I used the preferred leader election tool
> > for
> > > > all topics. Now, after finishing all of the broker restarts, the
> > cluster
> > > > seems to be stuck in leader election. Running the tool fails with
> > > > "kafka.admin.AdminOperationException: Preferred replica leader
> election
> > > > currently in progress..."
> > > >
> > > > Are any of these known issues? Is there a safer way to shutdown and
> > > restart
> > > > brokers that does not cause producer failures and under-replicated
> > > > partitions?
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message