kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Akhtar <ali.rac...@gmail.com>
Subject Re: Kafka null keys - OK or a problem?
Date Mon, 10 Oct 2016 06:51:21 GMT
Hey Michael,

We're using this one: https://github.com/Shopify/sarama

Any ideas how that one works?

On Mon, Oct 10, 2016 at 11:48 AM, Michael Noll <michael@confluent.io> wrote:

> FYI: Kafka's new Java producer (which ships with Kafka) the behavior is as
> follows:  If no partition is explicitly specified (to send the message to)
> AND the key is null, then the DefaultPartitioner [1] will assign messages
> to topic partitions in a round-robin fashion.  See the javadoc and also the
> little bit of code in [1] for details.
>
> Not sure which Go client you're using exactly so I can't tell whether your
> Go client follows the behavior of Kafka's Java producer.
>
> -Michael
>
>
>
>
> [1]
> https://github.com/apache/kafka/blob/trunk/clients/src/
> main/java/org/apache/kafka/clients/producer/internals/
> DefaultPartitioner.java
>
>
> On Mon, Oct 10, 2016 at 7:53 AM, Ali Akhtar <ali.rac200@gmail.com> wrote:
>
> > If keys are null, what happens in terms of partitioning, is the load
> spread
> > evenly..?
> >
> > On Mon, Oct 10, 2016 at 7:59 AM, Gwen Shapira <gwen@confluent.io> wrote:
> >
> > > Kafka itself supports null keys. I'm not sure about the Go client you
> > > use, but Confluent's Go client also supports null keys
> > > (https://github.com/confluentinc/confluent-kafka-go/).
> > >
> > > If you decide to generate keys and you want even spread, a random
> > > number generator is probably your best bet.
> > >
> > > Gwen
> > >
> > > On Sun, Oct 9, 2016 at 6:05 PM, Ali Akhtar <ali.rac200@gmail.com>
> wrote:
> > > > A kafka producer written elsewhere that I'm using, which uses the Go
> > > kafka
> > > > driver, is sending messages where the key is null.
> > > >
> > > > Is this OK - or will this cause issues due to partitioning not
> > happening
> > > > correctly?
> > > >
> > > > What would be a good way to generate keys in this case, to ensure
> even
> > > > partition spread?
> > > >
> > > > Thanks.
> > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter | blog
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message