kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Rann <br...@zendesk.com.INVALID>
Subject Re: Kafka consumer offset topic deletion
Date Wed, 19 Sep 2018 05:06:52 GMT
That's unusually large. Ours are around 32k-90mb each. Initially curious if
you
have log.cleaner.enable=true and what offsets.retention.minutes is set to.

And yes it can affect cluster performance. We had instances of consumer
outages
that were caused by bugged large consumer offfset files, especially during
broker restarts where that partition had an index repair, or during
leadership
changes, correlating with a large IO spike where presumably the whole
partition
is read.

for a given large partition, are the replicas the same size in each broker?
It's
possible for the log cleaner thread to die which needs a broker restart to
possibly fix, and investigation if it keeps dying. Do you see any errors in
your
logs around that topic-partition?

You can check (and monitor) the JMX metric for
kafka.log:type=LogCleanerManager,name=time-since-last-run-ms which if the
cleaner thread has died should be quite high. Our is normally around 15s.

ps: https://twitter.com/gwenshap/status/1026888706429353986

On Wed, Sep 19, 2018 at 6:25 AM Liam Clarke <liam.clarke@adscale.co.nz>
wrote:

> Odd that the log compaction isn't working. What OS is your broker running
> on and can you please post your server.properties?
>
> On Wed, 19 Sep. 2018, 2:13 am Kaushik Nambiar, <kaushiknambiar22@gmail.com
> >
> wrote:
>
> > >
> > > Hello,
> > > We have a Kafka 0.11.xx version setup.
> > > So the system topic which is __consumer_offset, we are looking at many
> > > such topics like __consumer_offset-1,2,4.....
> > > So one topic in particular,is now having log segments which is
> > > contributing to 5GB of data.
> > > I had a look at our server.properties file but couldnot find a property
> > > called offsets.retention.minutes (found via Google).
> > > So we are assuming Kafka will take the default value for such
> properties.
> > > So based on the default value of offset.retention.minutes property
> which
> > > is I guess 24 hours,but still we can see log segments older than few
> > month.
> > > So my question is ,is it safe to have that consumer_offset-7 topic
> > (having
> > > 5GB of log segments) deleted.Would it affect kafka anyway that would b
> a
> > > problem in production environment.
> > > If deleting the topics is not a safe implementation,can you suggest a
> > > different fix for this?
> > >
> > >
> > > Thanks and Regards,
> > > Kaushik Nambiar
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message