kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fran├žois Langelier <f.langel...@gmail.com>
Subject Re: Partition key not working properly
Date Wed, 26 Nov 2014 19:57:13 GMT
Hi haoming,

As far as I know, svante is right.

Maybe you modified your default partitioner?

or are you sure the same key go to different partitions? maybe its just 2
keys that are going to the same partition?

Because it's possible that you have something like that
- key "1" -> partition 3
- key "2" -> partition 2
- key "3" -> partition 3

and nothing in partition 1

On Wed Nov 26 2014 at 02:35:33 Haoming Zhang <haoming.zhang@outlook.com>
wrote:

> Hi Svante,
>
> Thanks for your reply!
>
> As you said, my purpose is let "all messages with the same key goes to the
> same partition", but the actual case is even I hard code the same partition
> key(let's say the key is "1") for three messages, the messages are still
> goes to different partitions.
>
> Regards,
> Haoming
>
> > Date: Wed, 26 Nov 2014 08:03:04 +0100
> > Subject: Re: Partition key not working properly
> > From: saka@csi.se
> > To: users@kafka.apache.org
> >
> > By default, the partition key is used for hashing then it's placed in a
> > partition that has the appropriate hashed keyspace.
> >
> > If you have three physical partitions and then give the partition key "5"
> > it has nothing to do with physical partition 5 (that does not exist) ,
> > similar to physical: partition = hash("5") mod 3
> >
> >
> > The only guarantee is that all messages with the same key goes to the
> same
> > partition. This is useful to make sure that for example all logs from the
> > same ip goest to the same partition which means that they can be read by
> > the same producer.
> >
> > /svante
> >
> >
> >
> > 2014-11-26 2:42 GMT+01:00 Haoming Zhang <haoming.zhang@outlook.com>:
> >
> > >
> > >
> > >
> > > Hi all,
> > >
> > > I'm struggling with how to use the partition key mechanism properly. My
> > > logic is set the partition number as 3, then  create three partition
> keys
> > > as "0", "1", "2", then use the partition keys to create three
> KeyedMessage
> > > such as
> > > KeyedMessage(topic, "0", message),
> > > KeyedMessage(topic, "1", message),
> > > KeyedMessage(topic, "2", message)
> > >
> > > After this, creating a producer instance to send out all the
> KeyedMessage.
> > >
> > > I expecting each KeyedMessage should enter to different partitions
> > > according to the different partition keys, which means
> > > KeyedMessage(topic, "0", message) go to Partition 0,
> > > KeyedMessage(topic, "1", message) go to Partition 1,
> > > KeyedMessage(topic, "2", message) go to Partition 2
> > >
> > > I'm using Kafka-web-console to watch the topic status, but the result
> is
> > > not like what I'm expecting. KeyedMessage still go to partitions
> randomly,
> > > some times two KeyedMessage will enter the same partition even they
> have
> > > different partition keys, .
> > >
> > > Not sure whether my logic is incorrect or I didn't understand the
> > > partition key mechanism correctly. Anyone could provides some sample
> code
> > > or explanation would be great!
> > >
> > > Thanks,
> > > Haoming
> > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message