kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: partitions stealing & balancing consumer threads across servers
Date Tue, 28 Oct 2014 01:21:02 GMT
You can take a look at the "consumer rebalancing algorithm" part in
http://kafka.apache.org/documentation.html. Basically, partitions are
evenly distributed among all consumers in the same group. If there are more
consumers in a group than partitions, some consumers will never get any
data.

Thanks,

Jun

On Mon, Oct 27, 2014 at 4:14 AM, Shlomi Hazan <shlomi@viber.com> wrote:

> Hi All,
>
> Using Kafka's high consumer API I have bumped into a situation where
> launching a consumer process P1 with X consuming threads on a topic with X
> partition kicks out all other existing consumer threads that consumed prior
> to launching the process P.
> That is, consumer process P is stealing all partitions from all other
> consumer processes.
>
> While understandable, it makes it hard to size & deploy a cluster with a
> number of partitions that will both allow balancing of consumption across
> consuming processes, dividing the partitions across consumers by setting
> each consumer with it's share of the total number of partitions on the
> consumed topic, and on the other hand provide room for growth and addition
> of new consumers to help with increasing traffic into the cluster and the
> topic.
>
> This stealing effect forces me to have more partitions then really needed
> at the moment, planning for future growth, or stick to what I need and
> trust the option to add partitions which comes with a price in terms of
> restarting consumers, bumping into out of order messages (hash
> partitioning) etc.
>
> Is this policy of stealing is intended, or did I just jump to conclusions?
> what is the way to cope with the sizing question?
>
> Shlomi
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message