kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Differences in size of data replicated by mirror maker
Date Fri, 23 Aug 2013 04:19:35 GMT
We have JMX beans that report #messages per topic? Does the total count
match btw the two clusters?

Thanks,

Jun


On Thu, Aug 22, 2013 at 2:14 PM, Rajasekar Elango <relango@salesforce.com>wrote:

> Hi,
>
> We are using mirrormaker to replicate data between two kafka clusters. I am
> seeing huge difference in size of log in data dir between the broker in
> source cluster vs broker in destination cluster:
>
> For eg: Size of ~/data/Topic-0/ is about 910 G in source broker, but only
> its only 25G in destination broker. I see segmented log files (~500 M) is
> created for about every 2 or 3 mins in source brokers, but I see segmented
> log files is created for about every 25 mins in destination broker.
>
> I verified mirrormaker is doing fine using consumer offset checker, not
> much lag, offsets are incrementing. I also verified that topics/partitions
> are not under replicated in both source and target cluster. What is the
> reason for this difference in disk usage?
>
>
> --
> Thanks,
> Raja.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message