samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bae, Jae Hyeon" <metac...@gmail.com>
Subject Re: KafkaCheckPointManager is too slow
Date Wed, 04 Nov 2015 06:01:51 GMT
I didn't know I have to manually set the checkpoint topic as
log-compaction. Is there any reason why log-compaction is not being set by
default on the checkpoint topic?

I will try and see how it works.

Thank you
Best, Jae

On Tue, Nov 3, 2015 at 8:33 PM, Yi Pan <nickpan47@gmail.com> wrote:

> Hi, Bae,
>
> Where did you see this log? Is it in JobRunner? Or AppMaster? Or
> SamzaContainer?
>
> There are a few factors that may have the impact:
> 1. How many system stream partitions you have as the input? And how many
> tasks are there?
> 2. Did you set your checkpoint topic as log-compact topic in Kafka? The
> topic size would be much smaller if log compaction is turned on.
>
> Regards
>
> -Yi
>
> On Tue, Nov 3, 2015 at 3:59 PM, Bae, Jae Hyeon <metacret@gmail.com> wrote:
>
> > Hi Samza Dev
> >
> > Do you know why the following job is taking too long?
> >
> > 2015-11-03 23:58:17 KafkaCheckpointManager [INFO] Get latest offset
> 3386930
> > for topic __samza_checkpoint_ver_1_for_xxx_1 and partition 0.
> >
> > This is seriously slowing down development. How can I fix this problem?
> >
> > Thank you
> > Best, Jae
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message