kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Crayford <tcrayf...@heroku.com>
Subject Re: rate-limiting on rebalancing, or sync from non-leaders?
Date Sun, 03 Jul 2016 22:37:42 GMT
Hi Charity,

I'm not sure about the roadmap. The way we (and linkedin/dropbox/netflix)
handle rebalances right now is to do a small handful of partitions at a
time (LinkedIn does 10 partitions at a time the last I heard), not a big
bang rebalance of all the partitions in the cluster. That's not perfect and
not great throttling, and I agree that it's something Kafka desperately
needs to work on.

Thanks

Tom Crayford
Heroku Kafka

On Sun, Jul 3, 2016 at 2:00 AM, Charity Majors <charity@hound.sh> wrote:

> Hi there,
>
> I'm curious if there's anything on the Kafka roadmap for adding
> rate-limiting or max-throughput for rebalancing processes.
>
> Alternately, if you have RF>2, maybe a setting to instruct followers to
> sync from other followers?
>
> I'm super impressed with how fast and efficient the kafka data rebalancing
> process is, but also fear for the future when it's battling for resources
> against high production trafffic.  :)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message