kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sathyakumar Seshachalam <sathyakumar_seshacha...@trimble.com>
Subject Process to enable log compaction on a cluster
Date Mon, 04 Jul 2016 06:08:55 GMT
Ok, Am in situation where all kafka nodes are going to run out of space.
This is because I had been running uncompacted __consumer_offset topic and
everything retained topics .

Now at a place, where I can afford to compact __consumer_offset topic and
also delete certain topics. I would like to know the right process to do
this.

Now since I am having close 1.8T of data on __consumer_offset topic and
more in the topics data, any log compaction and log deletion/trunction is
going to take time. Should I do this node by node. Will Kafka's replication
come in the way. (I have read that uncompacted data from the leader is sent
to the followers.)

Is there a clean process for this for a 3 node Kafka cluster ? Last time I
triggered a log compaction in all the 3 node simultaneously, all consumers
broke (I raised this in the same email group and got an answer to increase
the memory). Eventually they self-healed, but this caused some serious
disruption to the service, so before trying I want to make sure, there is a
cleaner process here.

Any help/pointers will be greatly appreciated.

Thanks,
Sathya

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message