kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: how to listen each partition with separate threads ?
Date Fri, 31 Mar 2017 17:03:24 GMT
That sounds a lot -- even if brokers can handle quite some load. How
many brokers do you got?

Main question: why do you need a consumer per partition? If you
processing super expensive such that one consumer cannot handle multiple
partitions?


-Matthias

On 3/31/17 5:02 AM, Laxmi Narayan wrote:
> Hi,
> 
> Is there any performance downside of creating so many consumers ?
> 
> I mean literally I am gonna create atleast 7k connections in that case , I
> have nearly 7k partitions  with a given topic.
> 
> 
> Keep learning keep moving .....
> 
> On Fri, Mar 31, 2017 at 12:48 PM, Matthias J. Sax <matthias@confluent.io>
> wrote:
> 
>> You need to create a KafkaConsumer per thread.
>>
>> -Matthias
>>
>> On 3/30/17 10:51 PM, Laxmi Narayan wrote:
>>> Hi ,
>>>
>>> I was thinking to listen each partition with separate thread in Kafka.
>>> But i get error saying :
>>>
>>>
>>>
>>>
>>> *org.apache.kafka.clients.consumer.KafkaConsumer@383ad023KafkaConsumer
>> is
>>> not safe for multi-threaded accessSubscription to topics, partitions and
>>> pattern are mutually exclusive.*​
>>>
>>>  Is there any way to achieve this ?
>>>
>>>
>>> *​​*
>>> Keep learning keep moving .....
>>>
>>
>>
> 


Mime
View raw message