kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 王国栋 <wangg...@gmail.com>
Subject Re: how to keep the same partition when using mirror maker
Date Wed, 03 Jul 2013 07:46:23 GMT
Thanks Jun.

Guodong


On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao <junrao@gmail.com> wrote:

> One of the issues is that in 0.7, the partitioning key is not stored on the
> broker. So, mirror maker won't know what key to use for partitioning.
>
> In 0.8, the partitioning key will be stored on the broker. Once kafka-957
> is resolved, mirror maker will be able to partition messages based on the
> same key in the target cluster.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jul 2, 2013 at 7:55 PM, 王国栋 <wanggd04@gmail.com> wrote:
>
> > HI guys,
> >
> > We are using kafka0.7.2, in our cluster, we use customized partition
> > functions in producer.
> > Say, we compute partition id with user id in our log.
> >
> > But when we use mirror maker to pull log, we find that mirror maker uses
> > random partition to push log into destination brokers.
> >
> > In my mind, we have to decompress logs from consumer and re-partition
> them
> > again. So, I am wondering if there are any good ways to keep partition
> rule
> > in mirror maker.
> >
> > Thanks,
> >
> > Guodong
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message