kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard L. Burton III" <mrbur...@gmail.com>
Subject Concurrency and KafkaConsumer?
Date Fri, 22 Apr 2016 00:33:52 GMT
I have a case in which I'm iterating over the results of a poll(..)
in KafkaConsumer. While I'm iterating over the records, the thread is
blocked. e.g., I'm writing to a BlockingQueue that's full.

Will this have any impact on the KafkaConsumer?

I have one more follow up question.

Let's say I have 20 topics. I plan to spawn 20 KafkaConsumer's to process
each topic because I'm looking to rate limit the speed at which the records
are processed to avoid overloading external systems.

The proper usage of KafkaConsumer in this case would be:


   - Create a thread with a new instance of KafkaConsumer used in it.
   - Ensure the group id for that consumer is unique
   - Ensure only one consumer for each topic is running to avoid
   re-balancing



-- 
-Richard L. Burton III
@rburton

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message