kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Compton <d...@danielcompton.net>
Subject Re: Elastsic Scaling
Date Fri, 21 Nov 2014 04:57:58 GMT
While it’s good to plan ahead for growth, Kafka will still let you add more partitions to
a topic https://kafka.apache.org/081/ops.html#basic_ops_modify_topic. This will rebalance
the hashing if you are partitioning by your key, and consumers will probably end up with different
partitions, but don’t feel like you have to make the perfect config right at the start.


> On 21/11/2014, at 5:44 pm, Joe Stein <joe.stein@stealth.ly> wrote:
> If you plan ahead of time with enough partitions then you won't fall into
> an issue of backed up consumers when you scale them up.
> If you have 100 partitions 20 consumers can read from them (each could read
> from 5 partitions). You can scale up to 100 consumers (one for each
> partition) as the upper limit. If you need more than that you should have
> had more than 100 partitions to start. Scaling down can go to 1 consumer if
> you wanted as 1 consumer can read from N partitions.
> If you are using the JVM you can look at
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> and
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> there
> are other options in other languages and in the JVM too
> https://cwiki.apache.org/confluence/display/KAFKA/Clients
> At the end of the day the Kafka broker will not impose any limitations for
> what you are asking currently (as per the wire protocol
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
> ) it is all about how the consumer is designed and developed.
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
> On Thu, Nov 20, 2014 at 3:18 PM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>> Hello,
>> We're looking into using Kafka for a improved version of a system and the
>> question of how to scale Kafka came up.  Specifically, we want to try to
>> make the system scale as transparently as possible.  The concern was that
>> if we go from N to N*2 consumers that we would have some that are still
>> backed up while the new ones were working on only some of the new records.
>> Also, if the load drops, can we scale down effectively?
>> I'm sure there's a way to do it.  I'm just hoping that someone has some
>> knowledge in this area.
>> Thanks.

View raw message