kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yury Ruchin <yuri.ruc...@gmail.com>
Subject Is Kafka documentation regarding null key misleading?
Date Fri, 05 Dec 2014 18:32:47 GMT
Hello,

I've come across a (seemingly) strange situation when my Kafka producer
gave so uneven distribution across partitions. I found that I used null key
to produce messages, guided by the following clause in the documentation:
"If the key is null, then a random broker partition is picked." However,
after looking at the code, I found that the broker partition is not truly
random for every message - instead, the randomly picked partition number
sticks and only refreshes after the topic.metadata.refresh.ms expires,
which is 10 minutes by default. So, with null key the producer keeps
writing to the same partition for 10 minutes.

Is my understanding of partitioning with null key correct? If yes,
shouldn't the documentation be fixed then to explicitly describe the sticky
pseudo-random partition assignment?

Thanks,
Yury

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message