kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Seshadri, Balaji" <Balaji.Sesha...@dish.com>
Subject RE: Single thread, Multiple partitions
Date Tue, 08 Apr 2014 19:26:38 GMT
We have 131 partitions and run 6 tomcat instances each spawning 5 threads.

Depending on the number of partitions you have you got to parallelize your consumers horizontally
to scale.

May be start with 10-20 consumer instances with 4-5 threads each processing more than one
partition might help.

20 instances  * 10 threads = 200

If you have 1000 partitions then distribution would be 5 partitions will be consumed by 1

This is just rough estimate based on my understanding.

-----Original Message-----
From: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) [mailto:skadambi@bloomberg.net] 
Sent: Tuesday, April 08, 2014 1:00 PM
To: Seshadri, Balaji; users@kafka.apache.org
Subject: RE: Single thread, Multiple partitions

Ah, thanks, figured it out now. 

What kind of bottlenecks should I expect to run into if I'm looking at 10s of 1000s of partitions
for a topic? The amount of data passing through each partition or in aggregate is somewhat
small (few 100 GB per day across all partitions). The high partition count is because it simplifies
application semantics.

----- Original Message -----
From: Balaji.Seshadri@dish.com
To: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN), users@kafka.apache.org
At: Apr  8 2014 14:08:41

I think you are looking for accessing messages from set of partitions by your own policy.You
should use simple consumers in 0.8 and maintain the offsets you have read.


If it is 0.9 I'm yet to come upto speed.



-----Original Message-----
From: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) [mailto:skadambi@bloomberg.net] 
Sent: Tuesday, April 08, 2014 11:58 AM
To: users@kafka.apache.org
Subject: Single thread, Multiple partitions

Let's say I've a single consumer thread reading off multiple partitions (I'll have around
10K partitions). As per the documentation on https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example,
there are no guarantees on the order in which messages are read off the set of partitions.
If I wanted to enforce priority-weighted round robin reads off the partitions, could I get
a pointer on what code to fiddle with? Thanks!

View raw message