kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vincenzo D'Amore" <v.dam...@gmail.com>
Subject Re: parallel processing of records in a Kafka consumer
Date Fri, 24 Nov 2017 09:36:36 GMT
Hi Matthias,

what you're writing is in sharp contrast with what I know...

I read that: "Kafka consumers are typically part of a consumer group. When
multiple consumers are subscribed to a topic and belong to the same
consumer group, each consumer in the group will receive messages from a
different subset of the partitions in the topic."

https://www.safaribooksonline.com/library/view/kafka-the-
definitive/9781491936153/ch04.html

This means that if there are four partition, first consumer within a
consumer group will read simultaneously from all the existing partitions.

If we add another consumer, the existing four partition will be divided
between the two consumers.

If we have more consumers than partitions, exceeding consumers will remain
idle.

https://gist.github.com/freedev/adc3e58789cc23d25d15a7d273535523

So, if I understood correctly, you cannot have multiple consumers *within
the same consumer group* concurrently consuming the same partition.

To be clear, I'm genuinely interested in understanding if I correctly get
how Kafka consumers works.
Comments and suggestions are welcome :)

Best regards,
Vincenzo

On Wed, Nov 22, 2017 at 11:15 PM, Matthias J. Sax <matthias@confluent.io>
wrote:

> I KafkaConsumer itself should be use single threaded. If you want to
> parallelize processing, each thread should have it's own KafkaConsumer
> instance and all consumers should use the same `group.id` in their
> configuration. Load will be shared over all running consumer
> automatically for this case.
>
>
> -Matthias
>
> On 11/22/17 12:22 PM, cours.systeme@gmail.com wrote:
> > I am testing a KafkaConsumer. How can I modify it to process records in
> parallel?
> >
>
>


-- 
Vincenzo D'Amore

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message