kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhavesh Mistry <mistry.p.bhav...@gmail.com>
Subject Uniform Distribution of Messages for Topic Across Partitions Without Effecting Performance
Date Tue, 05 Aug 2014 01:12:23 GMT
How to achieve uniform distribution of non-keyed messages per topic across
all partitions?

We have tried to do this uniform distribution across partition using custom
partitioning from each producer instance using round robing (
count(messages) % number of partition for topic). This strategy results in
very poor performance.  So we have switched back to random stickiness that
Kafka provide out of box per some interval ( 10 minutes not sure exactly )
per topic.

The above strategy results in consumer side lags sometime for some
partitions because we have some applications/producers  producing more
messages for same topic than other servers.

Can Kafka provide out of box uniform distribution by using coordination
among all producers and rely on measure rate such as  # messages per minute
or # of bytes produce per minute to achieve uniform distribution and
coordinate stickiness of partition among hundreds of producers for same
topic ?

Thanks,

Bhavesh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message