kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hans Jespersen <h...@confluent.io>
Subject Re: Regarding Kafka
Date Sun, 09 Oct 2016 04:43:49 GMT
Why do you have 10 topics?  It seems like you are treating topics like partitions and it's
unclear why you don't just have 1 topic with 10, 20, or even 30 partitions. Ordering is only
guaranteed at a partition level.

In general if you want to capacity plan for partitions you benchmark a single partition and
then divide your peak estimated throughput by the results of the single partition results.

If you expect the peak throughput to increase over time then double your partition count to
allow room to grow the number of consumers without having to repartition.

Sizing can be a bit more tricky if you are using keys but it doesn't sound like you are if
today you are publishing to topics the way you describe.

-hans

> On Oct 8, 2016, at 9:01 PM, Abhit Kalsotra <abhit011@gmail.com> wrote:
> 
> Guys any views ?
> 
> Abhi
> 
>> On Sat, Oct 8, 2016 at 4:28 PM, Abhit Kalsotra <abhit011@gmail.com> wrote:
>> 
>> Hello
>> 
>> I am using librdkafka c++ library for my application .
>> 
>> *My Kafka Cluster Set up*
>> 2 Kafka Zookeper running on 2 different instances
>> 7 Kafka Brokers , 4 Running on 1 machine and 3 running on other machine
>> Total 10 Topics and partition count is 3 with replication factor of 3.
>> 
>> Now in my case I need to be very specific for the *message order* when I
>> am consuming the messages. I know if all the messages gets produced to the
>> same partition, it always gets consumed in the same order.
>> 
>> I need expert opinions like what's the ideal partition count I should
>> consider without effecting performance.( I am looking for close to 100,000
>> messages per seconds).
>> The topics are from 0 to 9 and when I am producing messages I do something
>> like uniqueUserId % 10 , and then pointing to a respective topic like 0 ||
>> 1 || 2 etc..
>> 
>> Abhi
>> 
>> 
>> 
>> 
>> --
>> If you can't succeed, call it version 1.0
>> 
> 
> 
> 
> -- 
> If you can't succeed, call it version 1.0

Mime
View raw message