kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pritam Kadam <kpri...@thoughtworks.com>
Subject Kafka - Is it possible to achieve low latency (below 5ms?)
Date Sat, 02 Jun 2018 12:41:07 GMT
Hi,

I have been playing around with different kafka configurations at producer,
broker and consumer side.

But could not figure out a way to get predictable low latency below 5 to 10
ms.

*Use Case:*

   - Application has 100-200 producers, each producer is producing messages
   at different rates. Ex. 1 msg/sec, 10 msgs/sec, 100msgs/sec (100msg/sec is
   the max limit)
   - Each producer is producing messages on unique topic on a same
   partition so that ordering is maintained which is business requirement.
   - Every producer has corresponding consumers. Some producers might have
   1-10 consumers.

As I look at the above use case, I do not see I would be getting benefits
of batching which kafka is heavily depends on for getting maximum
throughput.

>From whole system I expect max throughput of around 40 - 50k but we would
like to achieve latency below 5-10 ms.

Is it possible with kafka to get this numbers of latency and throughput?

Note: I tried different configs focussing on latency part, but could not
get latency below 150ms, for example

kafka-clients {
  linger.ms=0
  acks=0
  batch.size=1
}

akka.kafka.consumer {
  poll-interval = 0
  poll-timeout = 0
}


Thanks,
Pritam.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message