kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Does kafka write key to broker?
Date Mon, 29 Jun 2015 20:22:43 GMT
Hi Mohit,

The KeyedMessage has a "key" and "partKey" as it fields, and if the partKey
is not specified then key will be used as partKey by default.

So for your case, if you do

new KeyedMessage[String, String](topic, null /*key*/, partkey, value)

then the partkey will be used to determine the partition but not written to
broker.

Guozhang

On Wed, Jun 24, 2015 at 9:45 AM, Mohit Kathuria <mkathuria@sprinklr.com>
wrote:

> Thanks Jason.
>
> Hi Liquan,
>
> I was doing this initially using KeyedMessage without setting the key. But
> if the key is null, the custom partitioner won't be used and kafka will
> write to random partition which is cached for some time within the
> producer. I have a custom partitioner to have fairness between the
> partitions.
>
> -Thanks,
> Mohit
>
> On Tue, Jun 23, 2015 at 10:34 PM, Liquan Pei <liquanpei@gmail.com> wrote:
>
> > Hi Mohit,
> >
> > If you instantiate the keyed message with
> > val topic = "topic"
> > val value = "value"
> > val message = new KeyedMessage[String, String](topic,  value);
> >
> > Then the key in the KeyedMessage will be null.
> >
> > Hope this helps!
> >
> > Thanks,
> > Liquan
> >
> > On Tue, Jun 23, 2015 at 8:18 AM, Mohit Kathuria <mkathuria@sprinklr.com>
> > wrote:
> >
> > > Hi,
> > >
> > > We are using kafka 0.8.1.1 in our production cluster. I recently
> started
> > > specifying key as the message itself. I just realised that the key is
> > also
> > > written to the broker which means that the data is duplicated within a
> > > keyed message. I am going to change the key. Stupid mistake.
> > >
> > > However, just out of anxiety, I want to know whether we can turn off
> > > writing the key to the broker. Any configuration I can change to
> achieve
> > > this?
> > >
> > > -Thanks,
> > > Mohit Kathuria
> > >
> >
> >
> >
> > --
> > Liquan Pei
> > Department of Physics
> > University of Massachusetts Amherst
> >
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message