kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koushik Chitta <kchi...@microsoft.com.INVALID>
Subject RE: Kakfa Broker scaling tips for high number of consumers for a single topic
Date Fri, 15 Nov 2019 00:31:49 GMT
What change did you observe in broker latency metrics 'totaltimems' ?


-----Original Message-----
From: aravind s <sumoo6649@gmail.com> 
Sent: Wednesday, November 13, 2019 11:03 PM
To: users@kafka.apache.org
Subject: Kakfa Broker scaling tips for high number of consumers for a single topic

Hi,

We have a use-case where there are close to 40 producers to a topic with 2 replicas and consumers
for this topic are around 2000. We have seen that producer latency goes by 3 times when consumers
grew from 500 to 2000. We changed the following properties. The machine has sufficient memory.
The rate is around 100k we are targeting.

   - queued.max.requests
   - num.io.threads
   - num.network.threads

But there is no impact. Please, could you suggest how to go about this?

Regards,
Aravind
Mime
View raw message