kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clark Breyman <cl...@breyman.com>
Subject Re: Controlled shutdown and leader election issues
Date Thu, 03 Apr 2014 19:01:02 GMT
Thank Neha - Is there a maven repo for pulling snapshot CI builds from?
Sorry if this is answered elsewhere.


On Wed, Apr 2, 2014 at 7:16 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> I'm not so sure if I know the issue you are running into but we fixed a few
> bugs with similar symptoms and the fixes are on the 0.8.1 branch. It will
> be great if you give it a try to see if your issue is resolved.
>
> Thanks,
> Neha
>
>
> On Wed, Apr 2, 2014 at 12:59 PM, Clark Breyman <clark@breyman.com> wrote:
>
> > Was there an answer for 0.8.1 getting stuck in preferred leader election?
> > I'm seeing this as well. Is there a JIRA ticket on this issue?
> >
> >
> > On Fri, Mar 21, 2014 at 1:15 PM, Ryan Berdeen <rberdeen@hubspot.com>
> > wrote:
> >
> > > So, for 0.8 without "controlled.shutdown.enable", why would
> > ShutdownBroker
> > > and restarting cause under-replication and producer exceptions? How
> can I
> > > upgrade gracefully?
> > >
> > > What's up with 0.8.1 getting stuck in preferred leader election?
> > >
> > >
> > > On Fri, Mar 21, 2014 at 12:18 AM, Neha Narkhede <
> neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > Which brings up the question - Do we need ShutdownBroker anymore? It
> > > seems
> > > > like the config should handle controlled shutdown correctly anyway.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Thu, Mar 20, 2014 at 9:16 PM, Jun Rao <junrao@gmail.com> wrote:
> > > >
> > > > > We haven't been testing the ShutdownBroker command in 0.8.1
> > rigorously
> > > > > since in 0.8.1, one can do the controlled shutdown through the new
> > > config
> > > > > "controlled.shutdown.enable". Instead of running the ShutdownBroker
> > > > command
> > > > > during the upgrade, you can also wait until under replicated
> > partition
> > > > > count drops to 0 after each restart before moving to the next one.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Thu, Mar 20, 2014 at 3:14 PM, Ryan Berdeen <
> rberdeen@hubspot.com>
> > > > > wrote:
> > > > >
> > > > > > While upgrading from 0.8.0 to 0.8.1 in place, I observed some
> > > > surprising
> > > > > > behavior using kafka.admin.ShutdownBroker. At the start, there
> were
> > > no
> > > > > > underreplicated partitions. After running
> > > > > >
> > > > > >   bin/kafka-run-class.sh kafka.admin.ShutdownBroker --broker
10
> ...
> > > > > >
> > > > > > Partitions that had replicas on broker 10 were under-replicated:
> > > > > >
> > > > > >   bin/kafka-topics.sh --describe --under-replicated-partitions
> ...
> > > > > >   Topic: analytics-activity Partition: 2  Leader: 12  Replicas:
> > 12,10
> > > > > Isr:
> > > > > > 12
> > > > > >   Topic: analytics-activity Partition: 6  Leader: 11  Replicas:
> > 11,10
> > > > > Isr:
> > > > > > 11
> > > > > >   Topic: analytics-activity Partition: 14 Leader: 14  Replicas:
> > 14,10
> > > > > Isr:
> > > > > > 14
> > > > > >   ...
> > > > > >
> > > > > > While restarting the broker process, many produce requests failed
> > > with
> > > > > > kafka.common.UnknownTopicOrPartitionException.
> > > > > >
> > > > > > After each broker restart, I used the preferred leader election
> > tool
> > > > for
> > > > > > all topics. Now, after finishing all of the broker restarts,
the
> > > > cluster
> > > > > > seems to be stuck in leader election. Running the tool fails
with
> > > > > > "kafka.admin.AdminOperationException: Preferred replica leader
> > > election
> > > > > > currently in progress..."
> > > > > >
> > > > > > Are any of these known issues? Is there a safer way to shutdown
> and
> > > > > restart
> > > > > > brokers that does not cause producer failures and
> under-replicated
> > > > > > partitions?
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message