kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu, Libo " <libo...@citi.com>
Subject RE: partition numbers and mirrormaker
Date Wed, 03 Jul 2013 16:15:39 GMT
Thanks, Jun. Previously I created the topic with the specified number of partitions on the
destination 
(target) cluster before I launched mirrormaker to do mirroring. But the mirrormaker only downloaded

data for some of partitions, and in some cases, it stopped randomly.  I will test this scenario
again with 
beta1.

Regards,

Libo


-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com] 
Sent: Wednesday, July 03, 2013 11:49 AM
To: users@kafka.apache.org
Subject: Re: partition numbers and mirrormaker

Yes, but you can change num.partitions in the target brokers.

Thanks,

Jun


On Wed, Jul 3, 2013 at 6:21 AM, Yu, Libo <libo.yu@citi.com> wrote:

> For a topic with N partitions on the source cluster, if I let 
> mirrormaker to create topic automatically, the same topic on the 
> destination cluster will have only one partition.
> Is that expected behavior?  Could you just give a yes/no answer? Thanks.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> Sent: Tuesday, July 02, 2013 5:23 PM
> To: users@kafka.apache.org
> Subject: Re: partition numbers and mirrormaker
>
> You can override the default number of partitions on the source and 
> target cluster to N. (num.partitions in the broker config). If you 
> need a higher (or lower) number of partitions for specific topics you 
> can override that on a per-topic basis (again, the config can be kept 
> identical between the source and target cluster).
>
> Thanks,
>
> Joel
>
> On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <libo.yu@citi.com> wrote:
> > Hi
> >
> > I asked this question previously and didn't get an answer.
> > If a topic has N partitions on source cluster, after being mirrored, 
> > on the destination cluster it has only one partition.
> >
> > To avoid this issue, before mirroring, I create the same topic with 
> > N partitions on the Destination cluster. But then the mirrormaker 
> > cannot fetch all partitions and stop randomly.
> >
> > So what should we expect from mirrormake and what is the right way 
> > to
> use it? Thanks.
> >
> >
> > Regards,
> >
> > Libo
> >
>

Mime
View raw message