kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN)" <skada...@bloomberg.net>
Subject RE: Single thread, Multiple partitions
Date Tue, 08 Apr 2014 19:00:08 GMT
Ah, thanks, figured it out now. 

What kind of bottlenecks should I expect to run into if I'm looking at 10s of 1000s of partitions
for a topic? The amount of data passing through each partition or in aggregate is somewhat
small (few 100 GB per day across all partitions). The high partition count is because it simplifies
application semantics.

----- Original Message -----
From: Balaji.Seshadri@dish.com
To: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN), users@kafka.apache.org
At: Apr  8 2014 14:08:41

I think you are looking for accessing messages from set of partitions by your own policy.You
should use simple consumers in 0.8 and maintain the offsets you have read.


If it is 0.9 I'm yet to come upto speed.



-----Original Message-----
From: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) [mailto:skadambi@bloomberg.net] 
Sent: Tuesday, April 08, 2014 11:58 AM
To: users@kafka.apache.org
Subject: Single thread, Multiple partitions

Let's say I've a single consumer thread reading off multiple partitions (I'll have around
10K partitions). As per the documentation on https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example,
there are no guarantees on the order in which messages are read off the set of partitions.
If I wanted to enforce priority-weighted round robin reads off the partitions, could I get
a pointer on what code to fiddle with? Thanks!

View raw message