samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yi Pan <nickpa...@gmail.com>
Subject Re: KafkaCheckPointManager is too slow
Date Wed, 04 Nov 2015 04:33:15 GMT
Hi, Bae,

Where did you see this log? Is it in JobRunner? Or AppMaster? Or
SamzaContainer?

There are a few factors that may have the impact:
1. How many system stream partitions you have as the input? And how many
tasks are there?
2. Did you set your checkpoint topic as log-compact topic in Kafka? The
topic size would be much smaller if log compaction is turned on.

Regards

-Yi

On Tue, Nov 3, 2015 at 3:59 PM, Bae, Jae Hyeon <metacret@gmail.com> wrote:

> Hi Samza Dev
>
> Do you know why the following job is taking too long?
>
> 2015-11-03 23:58:17 KafkaCheckpointManager [INFO] Get latest offset 3386930
> for topic __samza_checkpoint_ver_1_for_xxx_1 and partition 0.
>
> This is seriously slowing down development. How can I fix this problem?
>
> Thank you
> Best, Jae
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message