kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zijing Guo <alter...@yahoo.com.INVALID>
Subject kafka consumers parallel consuming at consumer level or thread level?
Date Wed, 04 Feb 2015 11:43:32 GMT
Hi,I have some question regarding how kafka consumers achieve parallel consuming for one topic.
Say I have 2 partitions for topic1 and I have a consumer Group A, now:1: If no consumer under
consumer Group A subscribe topic1, then no message will be delivery to this consumer group.2:
If there is only 1 consumer under consumer Group A that subscribe topic1, then this consumer
will consume all the data from topic1 that among the 2 partitions?3:in the High level consumer
example https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Exampleit says:
"if you provide more threads than there are partitions on the topic, some threads will never
see a message   
   - if you have more partitions than you have threads, some threads will receive data from
multiple partitions"
 so back to my example, If I create 2 threads under one consumer instance that I created
and will each thread corresponding to a specific partition under the topic1? what is the difference
that I create 2 consumers, each with only 1 thread under same consumer group and consume topic1?
Is kafka parallel consuming really down to the thread level that with 1 consumer instance
or multiple consumer instance, each with 1 thread?
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message