kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rosenberg <...@squareup.com>
Subject consumer rebalance weirdness
Date Thu, 07 Aug 2014 06:18:19 GMT
We've noticed that some of our consumers are more likely to repeatedly
trigger rebalancing when the app is consuming messages more slowly (e.g.
persisting data to back-end systems, etc.).

If on the other hand we 'fast-forward' the consumer (which essentially
means we tell it to consume but do nothing with the messages until all
caught up), it will never decide to do a rebalance during this time.  So it
can go hours without rebalancing while fast forwarding and consuming super
fast, while during normal processing, it might decide to rebalance every
minute or so.

Is there any simple explanation for this?

Usually the trigger for rebalance logged is that a "topic info for path X
has changed to Y, triggering rebalance".

Thanks for any ideas.

We'd like to reduce the rebalancing, as it essentially slows down
consumption each time it happens.

Thanks

Jason

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message