kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sharninder <sharnin...@gmail.com>
Subject Re: Horizontally Scaling Kafka Consumers
Date Thu, 30 Apr 2015 17:19:09 GMT
You need to first decide the conditions that need to be met for you to
scale to 50 consumers. These can be as simple as the consumer lag. Look at
the console offset checker tool and see if any of those numbers make sense.
Your existing consumers could also produce some metrics based on which
another process will decide when to spawn new customers.


On Wed, Apr 29, 2015 at 11:58 PM, Nimi Wariboko Jr <nimi@channelmeter.com>

> Hi,
> I was wondering what options there are/what other people are doing for
> horizontally scaling kafka consumers? Basically if I have 100 partitions
> and 10 consumers, and want to temporarily scale up to 50 consumers, what
> can I do?
> So far I've thought of just simply tracking consumer membership somehow
> (either through zookeeper's ephemeral nodes or maybe using gossip) on the
> consumers to achieve consensus on who consumes what. Another option would
> be having a router, possibly using something like nsq (I understand that
> they are similar pieces of software, but what we are going for is a
> persistent distributed queue (sharding) which is why I'm looking into
> Kafka)?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message