kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Stopford <...@confluent.io>
Subject Re: kafka compacted topic
Date Thu, 30 Nov 2017 09:40:30 GMT
The CPU/IO required to complete a compaction phase will grow as the log
grows but you can manage this via the cleaner's various configs. Check out
properties starting log.cleaner in the docs (
https://kafka.apache.org/documentation). All databases that implement LSM
storage have a similar overhead (Cassandra, HBase, RocksDB etc). Note the
first (active) segment is never compacted.

On Thu, Nov 30, 2017 at 6:59 AM Kane Kim <kane.isturm@gmail.com> wrote:

> I want to confirm if kafka has to re-compact all log segments, as log grows
> doesn't it become slower as well?
>
> On Tue, Nov 28, 2017 at 11:33 PM, Jakub Scholz <jakub@scholz.cz> wrote:
>
> > There is quite a nice section on this in the documentation -
> > http://kafka.apache.org/documentation/#compaction ... I think it should
> > answer your questions.
> >
> > On Wed, Nov 29, 2017 at 7:19 AM, Kane Kim <kane.isturm@gmail.com> wrote:
> >
> > > How does kafka log compaction work?
> > > Does it compact all of the log files periodically against new changes?
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message