kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: kafka consumers parallel consuming at consumer level or thread level?
Date Wed, 04 Feb 2015 16:04:56 GMT
Hi Edwin,

1. Yes.
2. Yes.
3. Yes; and there is no difference in terms of parallelism.

In the new consumer client
(org.apache.kafka.clients.consumer.KafkaConsumer) that is going to be out
in the next release, each consumer instance is single-threaded, i.e. it
will only have one thread for fetching data.

On Wed, Feb 4, 2015 at 3:43 AM, Zijing Guo <altergzj@yahoo.com.invalid>

> Hi,I have some question regarding how kafka consumers achieve parallel
> consuming for one topic. Say I have 2 partitions for topic1 and I have a
> consumer Group A, now:1: If no consumer under consumer Group A subscribe
> topic1, then no message will be delivery to this consumer group.2: If there
> is only 1 consumer under consumer Group A that subscribe topic1, then this
> consumer will consume all the data from topic1 that among the 2
> partitions?3:in the High level consumer example
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Exampleit
> says: "if you provide more threads than there are partitions on the topic,
> some threads will never see a message
>    - if you have more partitions than you have threads, some threads will
> receive data from multiple partitions"
>  so back to my example, If I create 2 threads under one consumer instance
> that I created and will each thread corresponding to a specific partition
> under the topic1? what is the difference that I create 2 consumers, each
> with only 1 thread under same consumer group and consume topic1? Is kafka
> parallel consuming really down to the thread level that with 1 consumer
> instance or multiple consumer instance, each with 1 thread?
> ThanksEdwin

-- Guozhang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message