kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Widman <j...@jeffwidman.com>
Subject Re: How to decommission a broker so the controller doesn't return it in the list of known brokers?
Date Fri, 09 Sep 2016 16:59:34 GMT
It looks like this problem is caused by this bug in Kafka 8, which was
fixed in Kafka 9:

https://issues.apache.org/jira/browse/KAFKA-972


On Thu, Sep 8, 2016 at 3:55 PM, Jeff Widman <jeff@jeffwidman.com> wrote:

> How do I permanently remove a broker from a Kafka cluster?
>
> Scenario:
>
> I have a stable cluster of 3 brokers. I temporarily added a fourth broker
> that successfully joined the cluster. The controller returned metadata
> indicating this broker was part of the cluster.
>
> However, I never rebalanced partitions onto this broker, so this broker #4
> was never actually used, didn't join any replica lists, ISRs, etc.
>
> I later decided to remove this unused broker from the cluster. I shutdown
> the broker successfully and Zookeeper /broker/ids no longer lists broker #4.
>
> However, when my application code connects to any Kafka broker and fetches
> metadata, I get a broker list that includes this deleted broker.
>
> How do I indicate to the cluster that this broker has been permanently
> removed from the cluster and not just a transient downtime?
>
> Additionally, what's happening under the covers that causes this?
>
> I'm guessing that when I connect to a broker and ask for metadata, the
> broker checks its local cache for the controller ID, contacts the broker
> and asks it for the list of all brokers. Then the controller checks it's
> cached list of brokers and returns the list of all brokers known to have
> belonged to the cluster at any point in time.
>
> I'm guessing this happens because it's not certain if the dead broker is
> permanently removed or just transient downtime. So I'm thinking I just need
> to indicate to the controller that it needs to reset it's list of known
> cluster brokers to the known live brokers in zookeeper. But would not be
> surprised if something in my mental model is incorrect.
>
> This is for Kafka 0.8.2. I am planning to upgrade to 0.10 in the
> not-to-distant future, so if 0.10 handles this differently, I'm also
> curious about that.
>
>
> --
>
> *Jeff Widman*
> jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
> <><
>



-- 

*Jeff Widman*
jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
<><

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message