kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chi Hoang <chi.ho...@zuora.com>
Subject Re: consumer_offsets partition skew and possibly ignored retention
Date Wed, 02 Nov 2016 21:07:21 GMT
I tried running reassignment on the topic, but that didn't help.  I had to
restart the broker for it to release the file handlers, then manually
delete.

On Fri, Oct 28, 2016 at 6:25 PM, James Brown <jbrown@easypost.com> wrote:

> I was having this problem with one of my __consumer_offsets partitions; I
> used reassignment to move the large partition onto a different set of
> machines (which forced the cleaner to run through them again) and after the
> new machines finished replicating, the partition was back down from 41GB to
> a nice trim 38MB.
>
> On Fri, Oct 28, 2016 at 1:00 PM, Chi Hoang <chi.hoang@zuora.com> wrote:
>
> > Hi,
> > We have a 3-node cluster that is running 0.9.0.1, and recently saw that
> the
> > "__consumer_offsets" topic on one of the nodes seems really skewed with
> > disk usage that looks like:
> >
> > 73G     ./__consumer_offsets-10
> > 0       ./__consumer_offsets-7
> > 0       ./__consumer_offsets-4
> > 0       ./__consumer_offsets-1
> > 0       ./__consumer_offsets-49
> > 19G     ./__consumer_offsets-46
> > 0       ./__consumer_offsets-43
> > 0       ./__consumer_offsets-40
> >
> >
> >
> > This goes on for all 50 partitions.  Upon inspection, we saw that a lot
> of
> > the log files were old:
> >
> > ll __consumer_offsets-10
> > total 76245192
> > -rw-r--r-- 1 root root         0 Oct  7 20:14 00000000000000000000.index
> > -rw-r--r-- 1 root root       901 Oct  7 20:14 00000000000000000000.log
> > -rw-r--r-- 1 root root    157904 Oct  7 22:15 00000000000907046457.index
> > -rw-r--r-- 1 root root 104855056 Oct  7 22:15 00000000000907046457.log
> > -rw-r--r-- 1 root root    157904 Oct  7 22:51 00000000000909543421.index
> > -rw-r--r-- 1 root root 104853568 Oct  7 22:51 00000000000909543421.log
> > -rw-r--r-- 1 root root    157904 Oct  7 23:27 00000000000910806717.index
> > -rw-r--r-- 1 root root 104853568 Oct  7 23:27 00000000000910806717.log
> >
> >
> > We are using default parameters as it pertains to offset management, and
> > our config output includes the following entries:
> >
> > log.cleaner.enable = true
> >
> > offsets.retention.minutes = 1440
> >
> >
> > I tried looking through the issues on JIRA but didn't see a reported
> > issue.  Does anyone know what's going on, and how I can fix this?
> >
> > Thanks.
> >
>
>
>
> --
> James Brown
> Engineer
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message