kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiaobing Bu <bushu...@gmail.com>
Subject kafka __conusmer_offsets cleanup.policy change from delete to compact
Date Thu, 22 Aug 2019 06:12:29 GMT
Hi,
Our kafka __consumer_offsets cleanup.policy set to delete, and retention.ms
set to 86400000(one day), because we found the default compact mode using
very large disk, and we change to delete mode many years ago.
Now we become understanding kafka failover, and found some partitions under
__conusmer_offsets size almost 28G(__consumer-offsets-12), and when a
broker shutdown manually or crash, another broker owned election and become
new partition leader, coordinator will read the 28G size offset and group
metadata, it will take about 4-5 mins, and between this time, our consumers
belong this coordinator can not commit the offsets, so can not consumer
messages.

So we want to change the cleanup.policy from delete to compact, and I test
this step on test environment, find the disk read speed almost 100M/s, and
I'm afraid this will take large disk press to our production kafka cluster,
And I want to consult can I delete some segment under __consumer_offsets-12
so , the compact segment will completed soon.

We not want to delete segment to anther __consumer_offsets partitions only
want to __consumer_offsets-12, so we don't want to using retention.ms to
small time.

Thanks.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message