kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Moritz Petersen <mpete...@adobe.com.INVALID>
Subject Best Practice Scaling Consumers
Date Mon, 06 May 2019 14:17:10 GMT
Hi all,

I’m new to Kafka and have a very basic question:

We build a cloud-scale platform and evaluate if we can use Kafka for pub-sub messaging between
our services. Most of our services scale dynamically based on load (number of requests, CPU
load etc.). In our current architecture, services are both, producers and consumers since
all services listen to some kind of events.

With Kafka, I assume we have two restrictions or issues:

  1.  Number of consumers is restricted to the number of partitions of a topic. Changing the
number of partitions is a relatively expensive operation (at least compared to scaling services).
Is it necessary to overprovision on the number of partitions in order to be prepared for load
peaks?
  2.  Adding or removing consumers halts processing of the related partition for a short period
of time. Is it possible to avoid or significantly minimize this lag?

Are there any additional best practices to implement Kafka consumers on a cloud scale environment?

Thanks,
Moritz

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message