kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Virendra Pratap Singh <vpsi...@yahoo-inc.com.INVALID>
Subject Uneven distribution of kafka topic partitions across multiple brokers
Date Tue, 24 Jun 2014 23:44:46 GMT
Have a kafka cluster with 10 brokers (kafka 0.8.0).  All of the brokers were setup upfront.
None was added later. Default number of partition is set to 4 and default replication to 2.
Have 3 topics in the system. None of these topics are manually created upfront, when the cluster
is setup. So relying on kafka to automatically create these topics when the producer(s) send
data first time for each of these topics.
We have multiple producer which will emit data for all of these topics at any point of time.
What it means is that kafka will be hit with producer request simultaneously from multiple
producer for producer request for these 3 topics.

What is observed is the topics partitions do not get spread out evenly in this scenario. There
are 10 brokers (ids 1-10) so expectation is that 3 * 4 = 12 topic partitions should be spread
out on all 10 servers. However in this case the first 2 brokers share most of the load and
few partitions are spread out. The same is true for the replicated instances also.

Here is the dump of list topic

topic: topic1  partition: 0    leader: 1       replicas: 1,2   isr: 1,2
topic: topic1  partition: 1    leader: 2       replicas: 2,1   isr: 2,1
topic: topic1  partition: 2    leader: 1       replicas: 1,2   isr: 1,2
topic: topic1  partition: 3    leader: 2       replicas: 2,1   isr: 2,1
topic: topic2        partition: 0    leader: 9       replicas: 9,4   isr: 9,4
topic: topic2        partition: 1    leader: 10      replicas: 10,5  isr: 10,5
topic: topic2        partition: 2    leader: 1       replicas: 1,6   isr: 1,6
topic: topic2        partition: 3    leader: 2       replicas: 2,7   isr: 2,7
topic: topic3     partition: 0    leader: 2       replicas: 2,1   isr: 2,1
topic: topic3     partition: 1    leader: 1       replicas: 1,2   isr: 1,2
topic: topic3     partition: 2    leader: 2       replicas: 2,1   isr: 2,1
topic: topic3     partition: 3    leader: 1       replicas: 1,2   isr: 1,2

So what is my options to have kafka evenly distribute the topic partitions? Would pre creating
the topics via create topic command help?

Regards,
Virendra

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message