kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clark Breyman <cl...@breyman.com>
Subject Re: Controlled shutdown and leader election issues
Date Fri, 04 Apr 2014 06:13:46 GMT
Done. https://issues.apache.org/jira/browse/KAFKA-1360


On Thu, Apr 3, 2014 at 9:13 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> >> Is there a maven repo for pulling snapshot CI builds from?
>
> We still need to get the CI build setup going, could you please file a JIRA
> for this?
> Meanwhile, you will have to just build the code yourself for now,
> unfortunately.
>
> Thanks,
> Neha
>
>
> On Thu, Apr 3, 2014 at 12:01 PM, Clark Breyman <clark@breyman.com> wrote:
>
> > Thank Neha - Is there a maven repo for pulling snapshot CI builds from?
> > Sorry if this is answered elsewhere.
> >
> >
> > On Wed, Apr 2, 2014 at 7:16 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > I'm not so sure if I know the issue you are running into but we fixed a
> > few
> > > bugs with similar symptoms and the fixes are on the 0.8.1 branch. It
> will
> > > be great if you give it a try to see if your issue is resolved.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Apr 2, 2014 at 12:59 PM, Clark Breyman <clark@breyman.com>
> > wrote:
> > >
> > > > Was there an answer for 0.8.1 getting stuck in preferred leader
> > election?
> > > > I'm seeing this as well. Is there a JIRA ticket on this issue?
> > > >
> > > >
> > > > On Fri, Mar 21, 2014 at 1:15 PM, Ryan Berdeen <rberdeen@hubspot.com>
> > > > wrote:
> > > >
> > > > > So, for 0.8 without "controlled.shutdown.enable", why would
> > > > ShutdownBroker
> > > > > and restarting cause under-replication and producer exceptions? How
> > > can I
> > > > > upgrade gracefully?
> > > > >
> > > > > What's up with 0.8.1 getting stuck in preferred leader election?
> > > > >
> > > > >
> > > > > On Fri, Mar 21, 2014 at 12:18 AM, Neha Narkhede <
> > > neha.narkhede@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Which brings up the question - Do we need ShutdownBroker anymore?
> > It
> > > > > seems
> > > > > > like the config should handle controlled shutdown correctly
> anyway.
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 20, 2014 at 9:16 PM, Jun Rao <junrao@gmail.com>
> wrote:
> > > > > >
> > > > > > > We haven't been testing the ShutdownBroker command in 0.8.1
> > > > rigorously
> > > > > > > since in 0.8.1, one can do the controlled shutdown through
the
> > new
> > > > > config
> > > > > > > "controlled.shutdown.enable". Instead of running the
> > ShutdownBroker
> > > > > > command
> > > > > > > during the upgrade, you can also wait until under replicated
> > > > partition
> > > > > > > count drops to 0 after each restart before moving to the
next
> > one.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 20, 2014 at 3:14 PM, Ryan Berdeen <
> > > rberdeen@hubspot.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > While upgrading from 0.8.0 to 0.8.1 in place, I observed
some
> > > > > > surprising
> > > > > > > > behavior using kafka.admin.ShutdownBroker. At the
start,
> there
> > > were
> > > > > no
> > > > > > > > underreplicated partitions. After running
> > > > > > > >
> > > > > > > >   bin/kafka-run-class.sh kafka.admin.ShutdownBroker
--broker
> 10
> > > ...
> > > > > > > >
> > > > > > > > Partitions that had replicas on broker 10 were
> > under-replicated:
> > > > > > > >
> > > > > > > >   bin/kafka-topics.sh --describe
> --under-replicated-partitions
> > > ...
> > > > > > > >   Topic: analytics-activity Partition: 2  Leader:
12
>  Replicas:
> > > > 12,10
> > > > > > > Isr:
> > > > > > > > 12
> > > > > > > >   Topic: analytics-activity Partition: 6  Leader:
11
>  Replicas:
> > > > 11,10
> > > > > > > Isr:
> > > > > > > > 11
> > > > > > > >   Topic: analytics-activity Partition: 14 Leader:
14
>  Replicas:
> > > > 14,10
> > > > > > > Isr:
> > > > > > > > 14
> > > > > > > >   ...
> > > > > > > >
> > > > > > > > While restarting the broker process, many produce
requests
> > failed
> > > > > with
> > > > > > > > kafka.common.UnknownTopicOrPartitionException.
> > > > > > > >
> > > > > > > > After each broker restart, I used the preferred leader
> election
> > > > tool
> > > > > > for
> > > > > > > > all topics. Now, after finishing all of the broker
restarts,
> > the
> > > > > > cluster
> > > > > > > > seems to be stuck in leader election. Running the
tool fails
> > with
> > > > > > > > "kafka.admin.AdminOperationException: Preferred replica
> leader
> > > > > election
> > > > > > > > currently in progress..."
> > > > > > > >
> > > > > > > > Are any of these known issues? Is there a safer way
to
> shutdown
> > > and
> > > > > > > restart
> > > > > > > > brokers that does not cause producer failures and
> > > under-replicated
> > > > > > > > partitions?
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message