kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Csar <christ...@csar.us>
Subject Re: Proper Relationship Between Partition and Threads
Date Wed, 28 Jan 2015 21:44:39 GMT
Ricardo,
   The parallelism of each logical consumer (consumer group) is the number
of partitions. So with four partitions it could make sense to have one
logical consumer (application) have two processes on different machines
each with two threads, or one process with four. While with two logical
consumers (two different applications) you would want each to have 4
threads (4*2 = 8 threads total).

There are also considerations depending on which consumer code you are
using (which I'm decidedly not someone with good information on)

Christian

On Wed, Jan 28, 2015 at 1:28 PM, Ricardo Ferreira <
jricardoferreira@gmail.com> wrote:

> Hi experts,
>
> I'm newbie in the Kafka world, so excuse me for such basic question.
>
> I'm in the process of designing a client for Kafka, and after few hours of
> study, I was told that to achieve a proper level of parallelism, it is a
> best practice having one thread for each partition of an topic.
>
> My question is that this rule-of-thumb also applies for multiple consumer
> applications. For instance:
>
> Considering a topic with 4 partitions, it is OK to have one consumer
> application with 4 threads, just like would be OK to have two consumer
> applications with 2 threads each. But what about having two consumer
> applications with 4 threads each? It would break any load-balancing made by
> Kafka brokers?
>
> Anyway, I'd like to understand if the proper number of threads that should
> match the number of partitions is per application or if there is some other
> best practice.
>
> Thanks in advance,
>
> Ricardo Ferreira
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message