kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Rann <br...@zendesk.com.INVALID>
Subject Re: Reduce number of brokers?
Date Tue, 18 Sep 2018 03:55:38 GMT
You need to do a partition reassignment to increase or decrease the
replication factor. It's tediously manual, but it's just json so it's
trivial to manipulate which is probably why it's still tediously manual.

There's a guide here although it's ageing a little:
http://www.allprogrammingtutorials.com/tutorials/changing-replication-factor-of-topic-in-kafka.php
and here:
https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor

You can use some built in tooling to help:

Create a topics.json file:

{"topics":
  [{"topic": "__consumer_offsets"}],
  "version":1
}

Assuming you currently have brokers 1,2,3,4 and you want to end up with 1,2:

bin/kafka-reassign-partitions.sh --zookeeper your_zk_string --generate
--topics-to-move-json-file topics.json --generate --broker-list 1,2,3

>From that you want to remove broker 3 from the reassignment proposal,
turning the 3 element array into 2.  If you're on a newer version of kafka
you also get log_dirs array in the output. For me that's always "any" so I
just take it out. If they're different for you you'll have more work to do.
If you don't want to edit it manually you could use some jq:

bin/kafka-reassign-partitions.sh --zookeeper your_zk_string --generate
--topics-to-move-json-file topics.json --generate --broker-list 1,2,3 | sed
-e '1,/Proposed/d' | jq 'del(.partitions[].replicas[] | select(. == 3)) |
del(.partitions[].log_dirs)' > reassignment.json

If you're happy with the reassignment file then execute it, running with
--verify instead of --execute to see when it's done.

bin/kafka-reassign-partitions.sh --zookeeper
your_zk_string --reassignment-json-file reassignment.json --execute

And then double check with:

bin/kafka-topics.sh --zookeeper your_zk_string --describe --topic
__consumer_offsets




On Tue, Sep 18, 2018 at 9:34 AM Dylan Martin <dmartin@istreamplanet.com>
wrote:

> I have a cluster with 4 brokers and I want to reduce it to 2 brokers.  I
> cannot re-assign __consumer_offsets because it wants at least 3 brokers.
>
>
> Is there a way to do this?  Or am I going to have to trash my cluster and
> start over?
>
>
> -Dylan
>
>
> (206) 855-9740 - Home
>
> (206) 235-8809 - Cell
>
> The information contained in this email message, and any attachment
> thereto, is confidential and may not be disclosed without the sender's
> express permission. If you are not the intended recipient or an employee or
> agent responsible for delivering this message to the intended recipient,
> you are hereby notified that you have received this message in error and
> that any review, dissemination, distribution or copying of this message, or
> any attachment thereto, in whole or in part, is strictly prohibited. If you
> have received this message in error, please immediately notify the sender
> by telephone, fax or email and delete the message and all of its
> attachments. Thank you.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message