kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: how to keep the same partition when using mirror maker
Date Wed, 03 Jul 2013 04:53:45 GMT
One of the issues is that in 0.7, the partitioning key is not stored on the
broker. So, mirror maker won't know what key to use for partitioning.

In 0.8, the partitioning key will be stored on the broker. Once kafka-957
is resolved, mirror maker will be able to partition messages based on the
same key in the target cluster.



On Tue, Jul 2, 2013 at 7:55 PM, 王国栋 <wanggd04@gmail.com> wrote:

> HI guys,
> We are using kafka0.7.2, in our cluster, we use customized partition
> functions in producer.
> Say, we compute partition id with user id in our log.
> But when we use mirror maker to pull log, we find that mirror maker uses
> random partition to push log into destination brokers.
> In my mind, we have to decompress logs from consumer and re-partition them
> again. So, I am wondering if there are any good ways to keep partition rule
> in mirror maker.
> Thanks,
> Guodong

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message