kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From VIPUL MALOO <vipul.ma...@microfocus.com>
Subject Regarding kafka memory not distributed across cluster
Date Wed, 08 May 2019 05:56:17 GMT
Hi all,

We are having a query regarding memory consumption on kafka scale out. It would be very helpful
if you can give suggestion/solution for the below query.
We are running kafka as docker container on kubernetes.

Memory limit of 4GiB is configured for Kafka broker POD. With some large load Kafka broker
POD's memory reached 4GiB. So we decided to manually scale out Kafka broker POD replicas from
1 to 3. But after scale out, for same load each Kafka broker PODs are consuming 4GiB memory.
 We expected Kafka broker POD's memory consumption to ~1.33GiB as we are running 3 PODs for
same amount of load.

Before Kafka Broker Scale out :
1 broker
6 topics each with 1 partition each
Memory consumption: 4GiB

After Kafka Broker Scale out and rebalancing topics over all the brokers:
3 broker
6 topics each with 1 partitions each
Memory consumption: 10GiB (Pod1: 2GiB, Pod2: 4GiB, Pod3: 4GiB)


After Kafka Broker Scale out and rebalancing topics over all the brokers:
3 broker
6 topics each with 3 partitions each
Memory consumption: 12GiB (Pod1: 4GiB, Pod2: 4GiB, Pod3: 4GiB)

All deployments are tested with same amount of load.

Regards,
Vipul Maloo
Micro Focus




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message