kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sathyakumar Seshachalam <sathyakumar_seshacha...@trimble.com>
Subject Re: Process to enable log compaction on a cluster
Date Mon, 04 Jul 2016 06:19:37 GMT
My another followup question is that Am I right in assuming that per topic
retention minutes or clean up policy, they all have any effect only when
you log.cleaner.enable-false ?
So In other words, I choose to truncate topic data, then __consumer_offsets
topic will also be either deleted or compacted ?

On Mon, Jul 4, 2016 at 11:38 AM, Sathyakumar Seshachalam <
sathyakumar_seshachalam@trimble.com> wrote:

> Ok, Am in situation where all kafka nodes are going to run out of space.
> This is because I had been running uncompacted __consumer_offset topic and
> everything retained topics .
>
> Now at a place, where I can afford to compact __consumer_offset topic and
> also delete certain topics. I would like to know the right process to do
> this.
>
> Now since I am having close 1.8T of data on __consumer_offset topic and
> more in the topics data, any log compaction and log deletion/trunction is
> going to take time. Should I do this node by node. Will Kafka's replication
> come in the way. (I have read that uncompacted data from the leader is sent
> to the followers.)
>
> Is there a clean process for this for a 3 node Kafka cluster ? Last time I
> triggered a log compaction in all the 3 node simultaneously, all consumers
> broke (I raised this in the same email group and got an answer to increase
> the memory). Eventually they self-healed, but this caused some serious
> disruption to the service, so before trying I want to make sure, there is a
> cleaner process here.
>
> Any help/pointers will be greatly appreciated.
>
> Thanks,
> Sathya
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message