kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerrit Jansen van Vuuren <gerrit...@gmail.com>
Subject Re: Kafka producer behavior
Date Tue, 17 Dec 2013 16:48:33 GMT
hi,

I've had the same issue with the kafka producer.

you need to use a different partitioner than the default one provided for
kafka.
I've created a round robin partitioner that works well for equally
distributing data across partitions.

https://github.com/gerritjvv/pseidon/blob/master/pseidon-kafka/java/pseidon/kafka/util/RoundRobinPartitioner.java





On Tue, Dec 17, 2013 at 5:32 PM, Hanish Bansal <
hanish.bansal.agarwal@gmail.com> wrote:

> Hi All,
>
> We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
> Replication Factor: 2
> Number of partitions: 2
>
> I have created a topic "test-topic1" in kafka.
>
> When i am listing status of that topic using bin/kafka-list-topic.sh, the
> status is:
>
> topic: test-topic1    partition: 0    leader: 0       replicas: 0,1   isr:
> 0,1
> topic: test-topic1    partition: 1    leader: 1       replicas: 1,0   isr:
> 1,0
>
> As both partition are on two separate nodes so when we produce the data it
> should be go to both nodes.
>
> But when i insert the data, it is going to only one node.
>
> For example if i insert 1000 messages then all 1000 messages will go either
> node1 or node2. Data is not evenly distributed on both nodes.
>
> Expected: 500 messages should go to node1 and 500 messages should go to
> node2.
>
> Any suggestion why i am facing this behavior?
>
> --
> *Thanks & Regards*
> *Hanish Bansal*
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message