kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakesh Vidyadharan <rvidyadha...@gracenote.com>
Subject How to work around log compaction error (
Date Wed, 27 Apr 2016 20:14:20 GMT

We enabled log compaction on a few topics, as we want to preserve permanently the latest versions
of messages published to specific topics.  After enabling compaction, the log cleaner thread
dies with the same error for the topics we tried it on.  It looks like kafka has starting
offset that does not exist in the topic (at least that is how I am reading the error).  Any
ideas on how we can work around this error?


[2016-04-27 15:52:11,306] INFO [kafka-log-cleaner-thread-0], Starting  (kafka.log.LogCleaner)
[2016-04-27 15:52:11,322] INFO Cleaner 0: Beginning cleaning of log metamorphosis.lineup-0.
[2016-04-27 15:52:11,323] INFO Cleaner 0: Building offset map for metamorphosis.lineup-0...
[2016-04-27 15:52:11,415] INFO Cleaner 0: Building offset map for log metamorphosis.lineup-0
for 1 segments in offset range [1553258, 2138466). (kafka.log.LogCleaner)
[2016-04-27 15:52:11,435] ERROR [kafka-log-cleaner-thread-0], Error due to  (kafka.log.LogCleaner)
java.lang.IllegalArgumentException: requirement failed: Last clean offset is 1553258 but segment
base offset is 2125968 for log metamorphosis.lineup-0.
at scala.Predef$.require(Predef.scala:233)
at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:509)
at kafka.log.Cleaner.clean(LogCleaner.scala:307)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:221)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:199)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
[2016-04-27 15:52:11,436] INFO [kafka-log-cleaner-thread-0], Stopped  (kafka.log.LogCleaner)

View raw message