kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <joe.st...@stealth.ly>
Subject Re: Uneven distribution of kafka topic partitions across multiple brokers
Date Wed, 25 Jun 2014 03:41:47 GMT
Take a look at

bin/kafka-reassign-partitions.sh

Option                                  Description

------                                  -----------

--broker-list <brokerlist>              The list of brokers to which the

                                          partitions need to be reassigned
in
                                          the form "0,1,2". This is
required
                                          if --topics-to-move-json-file is

                                          used to generate reassignment

                                          configuration

--execute                               Kick off the reassignment as
specified
                                          by the --reassignment-json-file

                                          option.

--generate                              Generate a candidate partition

                                          reassignment configuration. Note

                                          that this only generates a
candidate
                                          assignment, it does not execute
it.
--reassignment-json-file <manual        The JSON file with the partition

  assignment json file path>              reassignment configurationThe
format
                                          to use is -

                                        {"partitions":

                                        [{"topic": "foo",

                                          "partition": 1,

                                          "replicas": [1,2,3] }],

                                        "version":1

                                        }

--topics-to-move-json-file <topics to   Generate a reassignment
configuration
  reassign json file path>                to move the partitions of the

                                          specified topics to the list of

                                          brokers specified by the
--broker-
                                          list option. The format to use is
-
                                        {"topics":

                                        [{"topic": "foo"},{"topic":
"foo1"}],
                                        "version":1

                                        }

--verify                                Verify if the reassignment
completed
                                          as specified by the
--reassignment-
                                          json-file option.

--zookeeper <urls>                      REQUIRED: The connection string for

                                          the zookeeper connection in the
form
                                          host:port. Multiple URLS can be

                                          given to allow fail-over.

Command must include exactly one action: --generate, --execute or --verify

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/


On Tue, Jun 24, 2014 at 7:44 PM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:

> Have a kafka cluster with 10 brokers (kafka 0.8.0).  All of the brokers
> were setup upfront. None was added later. Default number of partition is
> set to 4 and default replication to 2.
> Have 3 topics in the system. None of these topics are manually created
> upfront, when the cluster is setup. So relying on kafka to automatically
> create these topics when the producer(s) send data first time for each of
> these topics.
> We have multiple producer which will emit data for all of these topics at
> any point of time. What it means is that kafka will be hit with producer
> request simultaneously from multiple producer for producer request for
> these 3 topics.
>
> What is observed is the topics partitions do not get spread out evenly in
> this scenario. There are 10 brokers (ids 1-10) so expectation is that 3 * 4
> = 12 topic partitions should be spread out on all 10 servers. However in
> this case the first 2 brokers share most of the load and few partitions are
> spread out. The same is true for the replicated instances also.
>
> Here is the dump of list topic
>
> topic: topic1  partition: 0    leader: 1       replicas: 1,2   isr: 1,2
> topic: topic1  partition: 1    leader: 2       replicas: 2,1   isr: 2,1
> topic: topic1  partition: 2    leader: 1       replicas: 1,2   isr: 1,2
> topic: topic1  partition: 3    leader: 2       replicas: 2,1   isr: 2,1
> topic: topic2        partition: 0    leader: 9       replicas: 9,4   isr:
> 9,4
> topic: topic2        partition: 1    leader: 10      replicas: 10,5  isr:
> 10,5
> topic: topic2        partition: 2    leader: 1       replicas: 1,6   isr:
> 1,6
> topic: topic2        partition: 3    leader: 2       replicas: 2,7   isr:
> 2,7
> topic: topic3     partition: 0    leader: 2       replicas: 2,1   isr: 2,1
> topic: topic3     partition: 1    leader: 1       replicas: 1,2   isr: 1,2
> topic: topic3     partition: 2    leader: 2       replicas: 2,1   isr: 2,1
> topic: topic3     partition: 3    leader: 1       replicas: 1,2   isr: 1,2
>
> So what is my options to have kafka evenly distribute the topic
> partitions? Would pre creating the topics via create topic command help?
>
> Regards,
> Virendra
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message