kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Controlled shutdown and leader election issues
Date Thu, 03 Apr 2014 02:16:49 GMT
I'm not so sure if I know the issue you are running into but we fixed a few
bugs with similar symptoms and the fixes are on the 0.8.1 branch. It will
be great if you give it a try to see if your issue is resolved.

Thanks,
Neha


On Wed, Apr 2, 2014 at 12:59 PM, Clark Breyman <clark@breyman.com> wrote:

> Was there an answer for 0.8.1 getting stuck in preferred leader election?
> I'm seeing this as well. Is there a JIRA ticket on this issue?
>
>
> On Fri, Mar 21, 2014 at 1:15 PM, Ryan Berdeen <rberdeen@hubspot.com>
> wrote:
>
> > So, for 0.8 without "controlled.shutdown.enable", why would
> ShutdownBroker
> > and restarting cause under-replication and producer exceptions? How can I
> > upgrade gracefully?
> >
> > What's up with 0.8.1 getting stuck in preferred leader election?
> >
> >
> > On Fri, Mar 21, 2014 at 12:18 AM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Which brings up the question - Do we need ShutdownBroker anymore? It
> > seems
> > > like the config should handle controlled shutdown correctly anyway.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Mar 20, 2014 at 9:16 PM, Jun Rao <junrao@gmail.com> wrote:
> > >
> > > > We haven't been testing the ShutdownBroker command in 0.8.1
> rigorously
> > > > since in 0.8.1, one can do the controlled shutdown through the new
> > config
> > > > "controlled.shutdown.enable". Instead of running the ShutdownBroker
> > > command
> > > > during the upgrade, you can also wait until under replicated
> partition
> > > > count drops to 0 after each restart before moving to the next one.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Thu, Mar 20, 2014 at 3:14 PM, Ryan Berdeen <rberdeen@hubspot.com>
> > > > wrote:
> > > >
> > > > > While upgrading from 0.8.0 to 0.8.1 in place, I observed some
> > > surprising
> > > > > behavior using kafka.admin.ShutdownBroker. At the start, there were
> > no
> > > > > underreplicated partitions. After running
> > > > >
> > > > >   bin/kafka-run-class.sh kafka.admin.ShutdownBroker --broker 10 ...
> > > > >
> > > > > Partitions that had replicas on broker 10 were under-replicated:
> > > > >
> > > > >   bin/kafka-topics.sh --describe --under-replicated-partitions ...
> > > > >   Topic: analytics-activity Partition: 2  Leader: 12  Replicas:
> 12,10
> > > > Isr:
> > > > > 12
> > > > >   Topic: analytics-activity Partition: 6  Leader: 11  Replicas:
> 11,10
> > > > Isr:
> > > > > 11
> > > > >   Topic: analytics-activity Partition: 14 Leader: 14  Replicas:
> 14,10
> > > > Isr:
> > > > > 14
> > > > >   ...
> > > > >
> > > > > While restarting the broker process, many produce requests failed
> > with
> > > > > kafka.common.UnknownTopicOrPartitionException.
> > > > >
> > > > > After each broker restart, I used the preferred leader election
> tool
> > > for
> > > > > all topics. Now, after finishing all of the broker restarts, the
> > > cluster
> > > > > seems to be stuck in leader election. Running the tool fails with
> > > > > "kafka.admin.AdminOperationException: Preferred replica leader
> > election
> > > > > currently in progress..."
> > > > >
> > > > > Are any of these known issues? Is there a safer way to shutdown and
> > > > restart
> > > > > brokers that does not cause producer failures and under-replicated
> > > > > partitions?
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message