kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth <srikanth...@gmail.com>
Subject Re: Hash partition of key with skew
Date Thu, 05 May 2016 20:36:18 GMT
Understood. Thanks.

On Wed, May 4, 2016 at 3:47 PM, Wesley Chow <wes@chartbeat.com> wrote:

>
> We don’t do this on the Kafka side, but for a different system that has
> similar distribution problems we manually maintain a map of “hot” keys. On
> the Kafka side, we distribute keys with an even distribution in our largest
> volume topic, and then squash the data and repartition based on a skewed
> key. The resulting skew is somewhat insignificant compared to our largest
> volume topic that we tend to not care.
>
> Wes
>
>
> > On May 4, 2016, at 2:57 PM, Srikanth <srikanth.ht@gmail.com> wrote:
> >
> > Yeah, fixed slicing may help. I'll put more thought into this.
> > You had mentioned that you didn't put custom partitioner into production.
> > Would you mind sharing how you worked around this currently?
> >
> > Srikanth
> >
> > On Tue, May 3, 2016 at 5:43 PM, Wesley Chow <wes@chartbeat.com> wrote:
> >
> >>>
> >>> Upload to S3 is partitioned by the "key" field. I.e, one folder per
> key.
> >> It
> >>> does offset management to make sure offset commit is in sync with S3
> >> upload.
> >>
> >> We do this in several spots and I wish we had built our system in such a
> >> way that we could just open source it. I’m sure many people have solved
> >> this repeatedly. We’ve had significant disk performance issues when the
> >> number of keys is large (40,000-ish in our case) — you can’t be
> expected to
> >> open a file per key. That’s why something like the fixed slicing
> strategy I
> >> described can make a big difference.
> >>
> >> Wes
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message