kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Levart <peter.lev...@gmail.com>
Subject Re: Doubts in Kafka
Date Thu, 10 Jan 2019 07:35:58 GMT
Hi Aruna,

On 1/10/19 8:19 AM, aruna ramachandran wrote:
> I am using keyed partitions with 1000 partitions, so I need to create 1000
> consumers because consumers groups and re balancing concepts is not worked
> in the case of manually assigned consumers.Is there any replacement for the
> above problem.

What API are you using in the KafkaConsumer? Are you using 
subscribe(Collection<String> topics) or are you using 
assign(Collection<TopicPartition> partitions) ?

The 1st one (subscribe) is the one you should be using for your usecase. 
With that call, when you subscribe to a multi-partition topic and you 
have multiple KafkaConsumer(s) configured with the same consumer group 
id, then partitions of the topic are dynamically assigned (and possibly 
reassigned when consumers come or go) to a set of live consumers. Will 
this work for you (and why not)?

Regards, Peter

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message