kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexis Midon <alexis.mi...@airbnb.com.INVALID>
Subject Re: rate-limiting on rebalancing, or sync from non-leaders?
Date Mon, 04 Jul 2016 12:14:56 GMT
Same here at Airbnb. Moving data is the biggest operational challenge
because of the network bandwidth cannibalization.
I was hoping that rate limiting would apply to replica fetchers too.

On Sun, Jul 3, 2016 at 15:38 Tom Crayford <tcrayford@heroku.com> wrote:

> Hi Charity,
>
> I'm not sure about the roadmap. The way we (and linkedin/dropbox/netflix)
> handle rebalances right now is to do a small handful of partitions at a
> time (LinkedIn does 10 partitions at a time the last I heard), not a big
> bang rebalance of all the partitions in the cluster. That's not perfect and
> not great throttling, and I agree that it's something Kafka desperately
> needs to work on.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Sun, Jul 3, 2016 at 2:00 AM, Charity Majors <charity@hound.sh> wrote:
>
> > Hi there,
> >
> > I'm curious if there's anything on the Kafka roadmap for adding
> > rate-limiting or max-throughput for rebalancing processes.
> >
> > Alternately, if you have RF>2, maybe a setting to instruct followers to
> > sync from other followers?
> >
> > I'm super impressed with how fast and efficient the kafka data
> rebalancing
> > process is, but also fear for the future when it's battling for resources
> > against high production trafffic.  :)
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message