kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damian Guy <damian....@gmail.com>
Subject Re: Struggling with Kafka Streams rebalances under load / in production
Date Sun, 11 Dec 2016 21:32:39 GMT
Hi Rob,

Do you have any further information you can provide? Logs etc?
Have you configured max.poll.interval.ms?

Thanks,
Damian

On Sun, 11 Dec 2016 at 20:30 Robert Conrad <robert@crunchbase.com> wrote:

> Hi All,
>
> I have a relatively complex streaming application that seems to struggle
> terribly with rebalance issues while under load. Does anyone have any tips
> for investigating what is triggering these frequent rebalances or
> particular settings I could experiment with to try to eliminate them?
>
> Originally I thought it had to do with exceeding the heartbeat timeout with
> heavy work threads, but the 0.10.1 release solved that by adding the
> background
> heartbeat thread
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread
> >.
> Now rebalance just seems to strike randomly and provide no insight into
> what triggered it (all nodes are happy, everything seems to be running
> smoothly).
>
> Any help or insight is greatly appreciated!
>
> Rob
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message