kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Consumer throughput and number of partitions
Date Wed, 11 Dec 2013 16:00:03 GMT
These numbers are a bit misleading. In Kafka, a topic partition is the
smallest unit that we distribute messages among consumers in the same
consumer group. So, if the number of consumers is larger than the total
number of partitions in a Kafka cluster, some consumers will never get any
data.

In your experiment with 2 partitions, only 2 of the consumers will get
data. The other 2 will not get any data. The 2 consumers that get data will
have a higher message rate. However, the aggregate consumption throughput
doesn't change (compared with the case with 10 partitions).

Thanks,

Jun


On Wed, Dec 11, 2013 at 12:38 AM, Arjun <arjun@socialtwist.com> wrote:

> Hi,
>
> I am trying my hands on kafka 0.8. I have 3 kafka servers and 3 zookeepers
> running.With the number of partitions as 10 and replication factor of 2, 4
> producers were pushing data into kafka, each has their own topic. There are
> 4 consumers which are getting the data from kafka.
>
> The problem i see is with the the number of partitions as 10, consumers
> are not consuming enough messages, the through put is almost all half, as
> supposed to when the number of partitions is 2.
>
> Is this a normal scenario or can we tune this behaviour so that the
> throughput will be high, when the number of partitions are high.
>
> environment 1 :
>     kafka servers : 3
>     Zookeepers : 3
>
>     producers : 4
>     consumers : 4
>     topics : 4
>
>     number of partitions: 10
>     replication factor : 2
>
>     number of messages pushed to kafka in 10 minutes : 47935 (per
> producer, all producers have pushed almost same number of messages)
>     number of messages consumed from kafka in same 10 minutes : 20693 (per
> consumer, all consumers have consumed almost same number of messages)
>
> environment 2 :
>     kafka servers : 3
>     Zookeepers : 3
>
>     producers : 4
>     consumers : 4
>     topics : 4
>
>     number of partitions: *2*
>     replication factor : 2
>
>     number of messages pushed to kafka in 10 minutes : 47935 (per
> producer, all producers have pushed almost same number of messages)
>     number of messages consumed from kafka in same 10 minutes : 47720 (per
> consumer, all consumers have consumed almost same number of messages)
>
>
> Thanks
> Arjun Narasimha Kota
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message