kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Rann <br...@zendesk.com.INVALID>
Subject Re: Kafka rebalancing behavior on broker failures
Date Mon, 23 Apr 2018 07:43:04 GMT
partitions are never automatically moved. They are assigned to broker(s)
and stay that way unless reassignments are triggered by external tools.
(leadership can move automatically though, if RF>1).

There's more info on partitions and moving partitions at these two links:
https://kafka.apache.org/documentation/#operations
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools

1) you need to do partition reassignments for the new broker to be involved
in existing topics.
2) no. it does leadership changes though if RF>1
3) if the data was deleted, when it comes back up with the same broker ID
if RF > 1 the partitions will automatically be synced back on to the
broker. I'm not sure with RF=1 but I guess it would be created as empty.
4) Gwen Shapira does a great talk on "when it absolutely, positively has to
be there" which covers failure scenarios:

https://www.confluent.io/resources/kafka-summit-2016/absolutely-positively-reliability-guarantees-kafka/

There's a 5 part series from Gwen here too:
https://www.confluent.io/online-talk/best-practices-for-apache-kafka-in-production-confluent-online-talk-series

And I believe part 2 "Reliability Guarantees in Ahache Kafka" is largely a
similar talk, and a bit more up to date. So maybe just go straight there.
And probably start with part 1. :)




On Mon, Apr 23, 2018 at 3:47 AM, Raghu Arur <raghuarur@gmail.com> wrote:

> I have a few questions on behavior of kafka w.r.t to broker life-cycle
> changes.
>
> 1. When a new broker is added to the cluster, do we need to manually invoke
> rebalancing or based on the load, kafka automatically invokes rebalancing ?
>
> 2. When an active broker goes down, does Kafka moves the partitions to
> other brokers ? And when the broker comes back again, does it get back the
> ownership of the partitions (or partition replicas) automatically without a
> manual rebalance ?
>
> 3. If a broker goes down and comes back up clean (for eg. disk went bad and
> the partitions got cleaned up) with the same broker ID, would the
> partitions be automatically moved to this broker or do we need to do a
> manual rebalance to get this broker handle some partitions ?
>
> 4. Is there a document/blog post/link that talks about Kafka broker
> behavior on failure scenarios.
>
> Thanks,
> Raghu.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message