kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathias Söderberg <mathias.soederb...@gmail.com>
Subject Re: Questions re: partition reassignment
Date Mon, 22 Sep 2014 20:32:59 GMT
>
> Partition reassignment will not move the leader unless the old leader is
>
not part of the new set of replicas.
> Even when it does move the leader, it waits until the new replicas enter
> the ISR.


Okay, so just to clarify, if I have a partition where the leader is broker
0, the ISR is [0, 1] and I make a partition reassignment with a new AR list
of [1, 0], broker 1 won't take over leadership? I was under the impression
that the "preferred" replica would become the leader, and that that would
be the first replica in the AR list?


> No. Kafka only waits until a replica enters the ISR set for it to be a
> candidate for leadership.


Alright, so if replica.lag.max.messages is set to the default of 4000, I
could potentially lose ~4000 when the leadership is transferred? (Assuming
it's a very high throughput topic/partition). Or what steps are actually
carried out when the leadership is transferred?


> can you share the logs? Is this truncation during the shutdown?


I'll see what I can dig up tomorrow, but it appears as if the broker that's
shutting down is resetting logs to some offset, which happens while it's
removing replica fetchers.

Best regards,
Mathias

On 22 September 2014 17:45, Neha Narkhede <neha.narkhede@gmail.com> wrote:

> My current interpretation is that if I start a partition reassignment, for
> the sake of simplicity let's assume it's just for a single partition, the
> new leader will first become a follower of the current leader, and when it
> has caught up it'll transfer leadership over to itself?
>
> Partition reassignment will not move the leader unless the old leader is
> not part of the new set of replicas.
> Even when it does move the leader, it waits until the new replicas enter
> the ISR.
>
> If so, is Kafka only concerned about data that has been flushed to disk? So
> if there's data that hasn't been flushed to disk yet it'll be discarded
> when the new leader gains leadership?
>
> No. Kafka only waits until a replica enters the ISR set for it to be a
> candidate for leadership.
>
> What happens if neither of the new ARs are part of the previous AR set? I
> suppose that they'll all be followers and when the new leader has caught up
> it'll assume leadership?
>
> Yes, this is the only time the leader is moved as part of partition
> reassignment.
>
> I've played around with controlled shutdown and noticed that the broker
> that's shutting down is truncating logs prior to actually shutting down,
> which I don't really understand?
>
> can you share the logs? Is this truncation during the shutdown?
>
> On Sun, Sep 21, 2014 at 2:32 PM, Mathias Söderberg <
> mathias.soederberg@gmail.com> wrote:
>
> > Good day,
> >
> > We're currently running a Kafka cluster in our staging environment,
> testing
> > everything out and so far it runs fairly smoothly.
> >
> > One thing that we're currently looking into is scaling a cluster that is
> > already up 'n serving data, and I have some questions regarding moving
> > partitions between brokers.
> >
> > My current interpretation is that if I start a partition reassignment,
> for
> > the sake of simplicity let's assume it's just for a single partition, the
> > new leader will first become a follower of the current leader, and when
> it
> > has caught up it'll transfer leadership over to itself?
> > If so, is Kafka only concerned about data that has been flushed to disk?
> So
> > if there's data that hasn't been flushed to disk yet it'll be discarded
> > when the new leader gains leadership?
> >
> > What happens if neither of the new ARs are part of the previous AR set? I
> > suppose that they'll all be followers and when the new leader has caught
> up
> > it'll assume leadership?
> >
> > I've played around with controlled shutdown and noticed that the broker
> > that's shutting down is truncating logs prior to actually shutting down,
> > which I don't really understand? I cannot really tell whether it does
> this
> > for all topics or just a subset of the ones that it's AR for as we have
> > quite a lot of topics, but I figured I'd just throw it out there, as I
> feel
> > like it could be explained by the answers to the questions above.
> >
> > Should mention that we're running v0.8.1.1 with mostly stock
> configuration,
> > and that our producers are currently using ACK = 1 as well.
> >
> > Best regards,
> > Mathias
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message