kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kaushik Nambiar <kaushiknambia...@gmail.com>
Subject Re: Kafka consumer offset topic deletion
Date Wed, 19 Sep 2018 06:34:25 GMT
Hello Bret,
Thank you for your reply.
For one consumer offset topic,I can see many log segments.Each log segment
is around 100 mb.
So due to many such log segments we are experiencing such data issues.
Your views on this one

Regards,
Kaushik Nambiar

On Wed, Sep 19, 2018, 10:37 AM Brett Rann <brann@zendesk.com.invalid> wrote:

> That's unusually large. Ours are around 32k-90mb each. Initially curious if
> you
> have log.cleaner.enable=true and what offsets.retention.minutes is set to.
>
> And yes it can affect cluster performance. We had instances of consumer
> outages
> that were caused by bugged large consumer offfset files, especially during
> broker restarts where that partition had an index repair, or during
> leadership
> changes, correlating with a large IO spike where presumably the whole
> partition
> is read.
>
> for a given large partition, are the replicas the same size in each broker?
> It's
> possible for the log cleaner thread to die which needs a broker restart to
> possibly fix, and investigation if it keeps dying. Do you see any errors in
> your
> logs around that topic-partition?
>
> You can check (and monitor) the JMX metric for
> kafka.log:type=LogCleanerManager,name=time-since-last-run-ms which if the
> cleaner thread has died should be quite high. Our is normally around 15s.
>
> ps: https://twitter.com/gwenshap/status/1026888706429353986
>
> On Wed, Sep 19, 2018 at 6:25 AM Liam Clarke <liam.clarke@adscale.co.nz>
> wrote:
>
> > Odd that the log compaction isn't working. What OS is your broker running
> > on and can you please post your server.properties?
> >
> > On Wed, 19 Sep. 2018, 2:13 am Kaushik Nambiar, <
> kaushiknambiar22@gmail.com
> > >
> > wrote:
> >
> > > >
> > > > Hello,
> > > > We have a Kafka 0.11.xx version setup.
> > > > So the system topic which is __consumer_offset, we are looking at
> many
> > > > such topics like __consumer_offset-1,2,4.....
> > > > So one topic in particular,is now having log segments which is
> > > > contributing to 5GB of data.
> > > > I had a look at our server.properties file but couldnot find a
> property
> > > > called offsets.retention.minutes (found via Google).
> > > > So we are assuming Kafka will take the default value for such
> > properties.
> > > > So based on the default value of offset.retention.minutes property
> > which
> > > > is I guess 24 hours,but still we can see log segments older than few
> > > month.
> > > > So my question is ,is it safe to have that consumer_offset-7 topic
> > > (having
> > > > 5GB of log segments) deleted.Would it affect kafka anyway that would
> b
> > a
> > > > problem in production environment.
> > > > If deleting the topics is not a safe implementation,can you suggest a
> > > > different fix for this?
> > > >
> > > >
> > > > Thanks and Regards,
> > > > Kaushik Nambiar
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message