kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Akhtar <ali.rac...@gmail.com>
Subject Re: Kafka null keys - OK or a problem?
Date Mon, 10 Oct 2016 06:58:27 GMT
It seems to be using a Hash Partitioner here:
https://github.com/Shopify/sarama/blob/master/config.go#L262

and HashPartitioner is documented as:

>  If the message's key is nil then a random partition is chosen

https://godoc.org/github.com/Shopify/sarama#example-Partitioner--Random

So.. it should be okay to have null keys, I'm guessing.

On Mon, Oct 10, 2016 at 11:51 AM, Ali Akhtar <ali.rac200@gmail.com> wrote:

> Hey Michael,
>
> We're using this one: https://github.com/Shopify/sarama
>
> Any ideas how that one works?
>
> On Mon, Oct 10, 2016 at 11:48 AM, Michael Noll <michael@confluent.io>
> wrote:
>
>> FYI: Kafka's new Java producer (which ships with Kafka) the behavior is as
>> follows:  If no partition is explicitly specified (to send the message to)
>> AND the key is null, then the DefaultPartitioner [1] will assign messages
>> to topic partitions in a round-robin fashion.  See the javadoc and also
>> the
>> little bit of code in [1] for details.
>>
>> Not sure which Go client you're using exactly so I can't tell whether your
>> Go client follows the behavior of Kafka's Java producer.
>>
>> -Michael
>>
>>
>>
>>
>> [1]
>> https://github.com/apache/kafka/blob/trunk/clients/src/main/
>> java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java
>>
>>
>> On Mon, Oct 10, 2016 at 7:53 AM, Ali Akhtar <ali.rac200@gmail.com> wrote:
>>
>> > If keys are null, what happens in terms of partitioning, is the load
>> spread
>> > evenly..?
>> >
>> > On Mon, Oct 10, 2016 at 7:59 AM, Gwen Shapira <gwen@confluent.io>
>> wrote:
>> >
>> > > Kafka itself supports null keys. I'm not sure about the Go client you
>> > > use, but Confluent's Go client also supports null keys
>> > > (https://github.com/confluentinc/confluent-kafka-go/).
>> > >
>> > > If you decide to generate keys and you want even spread, a random
>> > > number generator is probably your best bet.
>> > >
>> > > Gwen
>> > >
>> > > On Sun, Oct 9, 2016 at 6:05 PM, Ali Akhtar <ali.rac200@gmail.com>
>> wrote:
>> > > > A kafka producer written elsewhere that I'm using, which uses the
Go
>> > > kafka
>> > > > driver, is sending messages where the key is null.
>> > > >
>> > > > Is this OK - or will this cause issues due to partitioning not
>> > happening
>> > > > correctly?
>> > > >
>> > > > What would be a good way to generate keys in this case, to ensure
>> even
>> > > > partition spread?
>> > > >
>> > > > Thanks.
>> > >
>> > >
>> > >
>> > > --
>> > > Gwen Shapira
>> > > Product Manager | Confluent
>> > > 650.450.2760 | @gwenshap
>> > > Follow us: Twitter | blog
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message