kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonghui Zhao <zhaoyong...@gmail.com>
Subject Re: kafka deleted old logs but not released
Date Mon, 26 Jan 2015 03:57:44 GMT
I have fixed this issue like this patch
https://reviews.apache.org/r/29755/diff/5/.

I find rename failure still happens:

server.log.2015-01-26-06:[2015-01-26 06:10:54,513] ERROR File rename
failed, forcefully deleting file (kafka.log.Log)
server.log.2015-01-26-06:[2015-01-26 06:10:54,600] ERROR File rename
failed, forcefully deleting file (kafka.log.Log)
server.log.2015-01-26-06:[2015-01-26 06:10:54,685] ERROR File rename
failed, forcefully deleting file (kafka.log.Log)
server.log.2015-01-26-06:[2015-01-26 06:10:54,797] ERROR File rename
failed, forcefully deleting file (kafka.log.Log)
....

And use lsof I can still find some files opened by kafka but deleted,  but
those files sizes are 0.

java       3228 root   34uw     REG              253,2         0   26084228
/home/work/data/soft/kafka-0.8/data/.lock (deleted)
java       3228 root   35u      REG              253,2         0   26084232
/home/work/data/soft/kafka-0.8/data/cube-0/00000000000000000000.log
(deleted)
java       3228 root   36u      REG              253,2         0   26869778
/home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_topic_misearch_appstore-search-0/00000000000000003116.log
(deleted)
java       3228 root   37u      REG              253,2         0   26084234
/home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_topic_mishop-search_mishop_v1-0/00000000000000000000.log
(deleted)



Here is my configuration:

Binary: kafka_2.10-0.8.1.1
Retention config:

# The minimum age of a log file to be eligible for deletion
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log
as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new
log segment will be created.
log.segment.bytes=536870912

# The interval at which log segments are checked to see if they can be
deleted according
# to the retention policies
log.retention.check.interval.ms=60000

# By default the log cleaner is disabled and the log retention policy will
default to just delete segments after their retention expires.
# If log.cleaner.enable=true is set the cleaner will be enabled and
individual logs can then be marked for log compaction.
log.cleaner.enable=false



OS:  CentOS release 6.4 (Final)
JDK:
*java version "1.6.0_37"*
Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)

JDK is too old, but not sure if this result in rename failure.






2015-01-26 0:42 GMT+08:00 Jay Kreps <jay.kreps@gmail.com>:

> Also, what is the configuration for the servers? In particular it would be
> good to know the retention and/or log compaction settings as those delete
> files.
>
> -Jay
>
> On Sun, Jan 25, 2015 at 4:34 AM, Jaikiran Pai <jai.forums2013@gmail.com>
> wrote:
>
>> Hi Yonghui,
>>
>> Do you still have this happening? If yes, can you tell us a bit more
>> about your setup? Is there something else that accesses or maybe deleting
>> these log files? For more context to this question, please read the
>> discussion related to this here http://mail-archives.apache.
>> org/mod_mbox/kafka-dev/201501.mbox/%3C54C47E9B.5060401%40gmail.com%3E
>>
>>
>> -Jaikiran
>>
>>
>>> On Thursday 08 January 2015 11:19 AM, Yonghui Zhao wrote:
>>>
>>>> CentOS release 6.3 (Final)
>>>>
>>>>
>>>> 2015-01-07 22:18 GMT+08:00 Harsha <kafka@harsha.io>:
>>>>
>>>>  Yonghui,
>>>>>             Which OS you are running.
>>>>> -Harsha
>>>>>
>>>>> On Wed, Jan 7, 2015, at 01:38 AM, Yonghui Zhao wrote:
>>>>>
>>>>>> Yes  and I found the reason rename in deletion is failed.
>>>>>> In rename progress the files is deleted? and then exception blocks
>>>>>> file
>>>>>> closed in kafka.
>>>>>> But I don't know how can rename failure happen,
>>>>>>
>>>>>> [2015-01-07 00:10:48,685] ERROR Uncaught exception in scheduled task
>>>>>> 'kafka-log-retention' (kafka.utils.KafkaScheduler)
>>>>>> kafka.common.KafkaStorageException: Failed to change the log file
>>>>>> suffix
>>>>>> from  to .deleted for log segment 70781650
>>>>>>          at kafka.log.LogSegment.changeFileSuffixes(LogSegment.
>>>>>> scala:249)
>>>>>>          at kafka.log.Log.kafka$log$Log$$
>>>>>> asyncDeleteSegment(Log.scala:636)
>>>>>>          at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:627)
>>>>>>          at
>>>>>> kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:415)
>>>>>>          at
>>>>>> kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:415)
>>>>>>          at scala.collection.immutable.List.foreach(List.scala:318)
>>>>>>          at kafka.log.Log.deleteOldSegments(Log.scala:415)
>>>>>>          at
>>>>>>
>>>>>>  kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:325)
>>>>>
>>>>>
>>>>>>          at
>>>>>> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:356)
>>>>>>
>>>>>>          at
>>>>>> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:354)
>>>>>>
>>>>>>          at
>>>>>>
>>>>>>  scala.collection.TraversableLike$WithFilter$$
>>>>> anonfun$foreach$1.apply(TraversableLike.scala:772)
>>>>>
>>>>>>          at scala.collection.Iterator$class.foreach(Iterator.scala:
>>>>>> 727)
>>>>>>          at scala.collection.AbstractIterator.foreach(
>>>>>> Iterator.scala:1157)
>>>>>>          at
>>>>>> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>>>>>>          at scala.collection.AbstractIterable.foreach(
>>>>>> Iterable.scala:54)
>>>>>>          at
>>>>>>
>>>>>>  scala.collection.TraversableLike$WithFilter.
>>>>> foreach(TraversableLike.scala:771)
>>>>>
>>>>>>          at kafka.log.LogManager.cleanupLogs(LogManager.scala:354)
>>>>>>          at
>>>>>>
>>>>>>  kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:141)
>>>>>
>>>>>
>>>>>>          at
>>>>>> kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
>>>>>>          at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>>>>>>
>>>>>>          at
>>>>>>
>>>>>>  java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>>>
>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>>>>          at
>>>>>>
>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>> ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>>>>
>>>>>>          at
>>>>>>
>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>> ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>>>>
>>>>>>          at
>>>>>>
>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>>>>
>>>>>>          at
>>>>>>
>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.
>>>>> runTask(ThreadPoolExecutor.java:886)
>>>>>
>>>>>>          at
>>>>>>
>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>
>>>>>
>>>>>>          at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>>
>>>>>> 2015-01-07 13:56 GMT+08:00 Jun Rao <jun@confluent.io>:
>>>>>>
>>>>>>  Do you mean that the Kafka broker still holds a file handler on
a
>>>>>>>
>>>>>> deleted
>>>>>
>>>>>> file? Do you see those files being deleted in the Kafka log4j log?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Jun
>>>>>>>
>>>>>>> On Tue, Jan 6, 2015 at 4:46 AM, Yonghui Zhao <zhaoyonghui@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>  Hi,
>>>>>>>>
>>>>>>>> We use kafka_2.10-0.8.1.1 in our server. Today we found disk
space
>>>>>>>>
>>>>>>> alert.
>>>>>
>>>>>> We find many kafka data files are deleted, but still opened by kafka.
>>>>>>>>
>>>>>>>> such as:
>>>>>>>>
>>>>>>>> _yellowpageV2-0/00000000000068170670.log (deleted)
>>>>>>>> java       8446         root  724u      REG 253,2
>>>>>>>>
>>>>>>> 536937911
>>>>>
>>>>>> 26087362
>>>>>>>>
>>>>>>>>
>>>>>>>>  /home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_
>>>>> topic_ypgsearch_yellowpageV2-0/00000000000068818668.log
>>>>>
>>>>>> (deleted)
>>>>>>>> java       8446         root  725u      REG 253,2
>>>>>>>>
>>>>>>> 536910838
>>>>>
>>>>>> 26087364
>>>>>>>>
>>>>>>>>
>>>>>>>>  /home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_
>>>>> topic_ypgsearch_yellowpageV2-0/00000000000069457098.log
>>>>>
>>>>>> (deleted)
>>>>>>>> java       8446         root  726u      REG 253,2
>>>>>>>>
>>>>>>> 536917902
>>>>>
>>>>>> 26087368
>>>>>>>>
>>>>>>>>
>>>>>>>>  /home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_
>>>>> topic_ypgsearch_yellowpageV2-0/00000000000070104914.log
>>>>>
>>>>>> (deleted)
>>>>>>>>
>>>>>>>>
>>>>>>>> Is there anything wrong or wrong configed?
>>>>>>>>
>>>>>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message