kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ricardo Ferreira <rifer...@riferrei.com>
Subject Re: kafka log retention problem
Date Mon, 15 Jun 2020 17:44:48 GMT
It sounds like you are trying to forcibly delete the files that build up 
the segments used by the partitions. If that is the case then I would 
recommend not using external tools and leave to Kafka manage its 
filesystem. If you set the retention policy (either by size or time) in 
your topics the log cleaner threads will take care of purging any unused 
space from the segments automatically. No need to have a external 
process that clean those out.

In order to recover from this; kill any processes that might be acting 
upon those files and then restart your Kafka brokers once again. Only 
the broker JVM should have file handles to the segments to avoid any 
lock problems during file write/read.

Thanks,

-- Ricardo

On 6/15/20 4:50 AM, Ncir, Kawther (external - Service) wrote:
> Hi,
> I am working on a project using kafka broker and I face a problem with kafka log retention
,
> We need to delete logs every 24h  but  after the retention time the broker was stopped
and make an issue like this :
> "ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
> org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segme
> nts for test-1 in dir C:\tmp\log Caused by: java.nio.file.FileSystemException: C:\tmp\log\test-1\0000000000000000
> 0000.timeindex -> C:\tmp\log\test-1\00000000000000000000.timeindex.deleted: The process
cannot access the file because it is being used by another process."
>
> Could you please support me to find a solution.
>
> Regards,
> kawther
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message