kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rosenberg <...@squareup.com>
Subject Re: How to achieve distributed processing and high availability simultaneously in Kafka?
Date Wed, 06 May 2015 12:48:10 GMT
A consumer thread can consume multiple partitions.  This is not unusual, in
practice.

In the example you gave, if multiple high-level consumers are using the
same group id, they will automatically rebalance the partition assignment
between them as consumers dynamically join and leave the group.  So, in
your example, if process 1 dies, then process 2 will assume ownership for
all the n partitions (and if it has n/2 threads, each thread will own 2 of
the partitions).

In my experience though, its generally fine to have fewer threads than
partitions.  It depends on the volume of data incoming to each partition of
course, and how fast the consumer takes to process each message.

Jason

On Wed, May 6, 2015 at 1:57 AM, sumit jain <sumitjain91@gmail.com> wrote:

> I have a topic consisting of n partitions. To have distributed processing I
> create two processes running on different machines. They subscribe to the
> topic with same groupd id and allocate n/2 threads, each of which processes
> single stream(n/2 partitions per process).
>
> With this I will have achieved load distribution, but now if process 1
> crashes, than process 2 cannot consume messages from partitions allocated
> to process 1, as it listened only on n/2 streams at the start.
>
> Or else, if I configure for HA and start n threads/streams on both
> processes, then when one node fails, all partitions will be processed by
> other node. But here, we have compromised distribution, as all partitions
> will be processed by a single node at a time.
>
> Is there a way to achieve both simultaneously and how?
> Note: Already asked this on stackoverflow
>
> http://stackoverflow.com/questions/30060261/how-to-achieve-distributed-processing-and-high-availability-simultaneously-in-ka
> .
> --
> Thanks & Regards,
> Sumit Jain
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message