kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rosenberg <...@squareup.com>
Subject Re: consumer rebalance weirdness
Date Thu, 07 Aug 2014 15:15:37 GMT
Well, it's possible that when processing, it might take longer than the
zookeeper timeout to process a message, intermittently.  Would that cause a
zookeeper timeout?

(btw I'm usind 0.8.1.1).


On Thu, Aug 7, 2014 at 2:30 AM, Clark Haskins <chaskins@linkedin.com.invalid
> wrote:

> Is your application possibly timing out its zookeeper connection during
> consumption while doing its processing, thus triggering the rebalance?
>
> -Clark
>
> On 8/6/14, 11:18 PM, "Jason Rosenberg" <jbr@squareup.com> wrote:
>
> >We've noticed that some of our consumers are more likely to repeatedly
> >trigger rebalancing when the app is consuming messages more slowly (e.g.
> >persisting data to back-end systems, etc.).
> >
> >If on the other hand we 'fast-forward' the consumer (which essentially
> >means we tell it to consume but do nothing with the messages until all
> >caught up), it will never decide to do a rebalance during this time.  So
> >it
> >can go hours without rebalancing while fast forwarding and consuming super
> >fast, while during normal processing, it might decide to rebalance every
> >minute or so.
> >
> >Is there any simple explanation for this?
> >
> >Usually the trigger for rebalance logged is that a "topic info for path X
> >has changed to Y, triggering rebalance".
> >
> >Thanks for any ideas.
> >
> >We'd like to reduce the rebalancing, as it essentially slows down
> >consumption each time it happens.
> >
> >Thanks
> >
> >Jason
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message