kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: "support" topics for KStreams
Date Mon, 19 Dec 2016 21:58:25 GMT
For changelog topics, their retention policy is based on the log compaction
by default, and users can customize it if necessary when creating the state
store (this is in trunk, not in released versions yet).

For repartition topics, their retention policy is based on whatever the
broker's default retention policies.

I suspect your main issue of relentless rebalancing is due to state stores
being restoring when one or more instances are rolling bounced, which
requires replaying the state store changelog topic from scratch. So if your
changelog topic is very long, it will take very long time to restore.

Guozhang

On Thu, Dec 15, 2016 at 12:54 PM, Jon Yeargers <jon.yeargers@cedexis.com>
wrote:

> What's the retention settings for these (<topic>-changelog and
> -replication)? Im wondering about the relentless rebalancing issues Im
> facing and wondering if it has anything to do with consumers that lag too
> far behind.
>
> If I delete all the topics associated with a KStream project and restart it
> there are no rebalance issues. Everything is fast and responsive.
>
> Over the course of 6-10 hours of execution the rebalances take longer and
> longer until eventually the app(s) stop responding at all.
>
> Just curious.
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message