kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clark Haskins <chask...@linkedin.com.INVALID>
Subject Re: consumer rebalance weirdness
Date Thu, 07 Aug 2014 06:30:25 GMT
Is your application possibly timing out its zookeeper connection during
consumption while doing its processing, thus triggering the rebalance?

-Clark

On 8/6/14, 11:18 PM, "Jason Rosenberg" <jbr@squareup.com> wrote:

>We've noticed that some of our consumers are more likely to repeatedly
>trigger rebalancing when the app is consuming messages more slowly (e.g.
>persisting data to back-end systems, etc.).
>
>If on the other hand we 'fast-forward' the consumer (which essentially
>means we tell it to consume but do nothing with the messages until all
>caught up), it will never decide to do a rebalance during this time.  So
>it
>can go hours without rebalancing while fast forwarding and consuming super
>fast, while during normal processing, it might decide to rebalance every
>minute or so.
>
>Is there any simple explanation for this?
>
>Usually the trigger for rebalance logged is that a "topic info for path X
>has changed to Y, triggering rebalance".
>
>Thanks for any ideas.
>
>We'd like to reduce the rebalancing, as it essentially slows down
>consumption each time it happens.
>
>Thanks
>
>Jason


Mime
View raw message