kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Rischmann <vinc...@rischmann.fr>
Subject Spread log segment deletion over a couple hours
Date Wed, 02 May 2018 18:52:50 GMT
Hi,

I'm wondering if there is a way to tell Kafka to spread the log file
deletion when decreasing the retention time of a topic, and if not, if
it would make sense.
I'm asking because this afternoon, after decreasing the retention time
from 2 months to 1 month on 4 of my topics, the whole cluster became
overloaded for approximately 15 minutes (every broker with 25+ load,
disk usage almost 100%), with leader reelection, under replicated
partitions, and a bunch of consumers unable to make progress.
The change removed 5Tib of data across the 4 topics and I didn't check
beforehand to make sure how it would affect disk i/o, so it's on me that
this happened, but seeing how much data was removed I think it would
make sense to delete only a couple segments at a time in order to not
overload the disks.
Right now I can only be careful and plan the decrease in small steps but
that's going to be a little tedious.
How does everyone deal with this ?

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message