kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Michalski <michal.michal...@boxever.com>
Subject Re: Is Kafka documentation regarding null key misleading?
Date Fri, 05 Dec 2014 18:43:09 GMT
Yes, it is *very* misleading in my opinion - I've seen so many people
surprised with that behaviour...
Technically it's 100% correct of course: "If the key is null, then the
Producer will assign the message to a random Partition." - that's what
actually happens, because assignment is random. However, everyone I know
(including myself) assumed that "random" means "random for each message",
rather than "picked randomly every topic.metadata.refresh.ms".
So no, you're not the only one who got that wrong ;-)

M.

Kind regards,
MichaƂ Michalski,
michal.michalski@boxever.com

On 5 December 2014 at 18:32, Yury Ruchin <yuri.ruchin@gmail.com> wrote:

> Hello,
>
> I've come across a (seemingly) strange situation when my Kafka producer
> gave so uneven distribution across partitions. I found that I used null key
> to produce messages, guided by the following clause in the documentation:
> "If the key is null, then a random broker partition is picked." However,
> after looking at the code, I found that the broker partition is not truly
> random for every message - instead, the randomly picked partition number
> sticks and only refreshes after the topic.metadata.refresh.ms expires,
> which is 10 minutes by default. So, with null key the producer keeps
> writing to the same partition for 10 minutes.
>
> Is my understanding of partitioning with null key correct? If yes,
> shouldn't the documentation be fixed then to explicitly describe the sticky
> pseudo-random partition assignment?
>
> Thanks,
> Yury
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message