kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: Horizontally Scaling Kafka Consumers
Date Wed, 29 Apr 2015 12:38:14 GMT
Please correct me if wrong, but I think it is really not hard constraint
that one cannot have more consumers (from same group) than partitions on
single topic - all the surplus consumers will not be assigned to consume
any partition, but they can be there and as soon as one active consumer
from same group goes offline (its connection to ZK is dropped), consumers
from the group will be rebalanced so one passively waiting consumer will
become active.

Kind regards,
Stevo Slavic.

On Wed, Apr 29, 2015 at 2:25 PM, David Corley <davidcorley@gmail.com> wrote:

> If the 100 partitions are all for the same topic, you can have up to 100
> consumers working as part of a single consumer group for that topic.
> You cannot have more consumers than there are partitions within a given
> consumer group.
>
> On 29 April 2015 at 08:41, Nimi Wariboko Jr <nimi@channelmeter.com> wrote:
>
> > Hi,
> >
> > I was wondering what options there are for horizontally scaling kafka
> > consumers? Basically if I have 100 partitions and 10 consumers, and want
> to
> > temporarily scale up to 50 consumers, what options do I have?
> >
> > So far I've thought of just simply tracking consumer membership somehow
> > (either through Raft or zookeeper's znodes) on the consumers.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message