kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Manually deleting log files
Date Thu, 06 Oct 2011 20:39:41 GMT
Carlo,

You can delete those log files when the broker is still running since the
broker keeps an open file descriptor on each file. The only way to delete
those files is to set the proper config. We now have retention by either
time or size.

Jun

On Thu, Oct 6, 2011 at 12:25 PM, Carlo Cabanilla <carlo@datadoghq.com>wrote:

>  Hi,
>
> Is there a clean way to delete kafka logs that haven't expired yet? When I
> try to just delete the files from the file system, Kafka is still holding
> onto them so it doesn't free up space. From lsof:
>
>
> java 4747 kafka 152u  REG  8,16 537008063 2252888
> /mnt/kafka-logs/dispatch-0/00000000045637816846.kafka (deleted)
> java 4747 kafka 153u  REG  8,16 537039735 2252889
> /mnt/kafka-logs/dispatch-0/00000000046174824909.kafka (deleted)
> java 4747 kafka 154u  REG  8,16 536889360 2252890
> /mnt/kafka-logs/dispatch-0/00000000046711864644.kafka (deleted)
> java 4747 kafka 155u  REG  8,16 536909503 2252891
> /mnt/kafka-logs/dispatch-0/00000000047248754004.kafka (deleted)
> java 4747 kafka 156u  REG  8,16 536879126 2252892
> /mnt/kafka-logs/dispatch-0/00000000047785663507.kafka (deleted)
> java 4747 kafka 157u  REG  8,16 536904918 2252893
> /mnt/kafka-logs/dispatch-0/00000000048322542633.kafka (deleted)
> java 4747 kafka 158u  REG  8,16 536959487 2252894
> /mnt/kafka-logs/dispatch-0/00000000048859447551.kafka (deleted)
> java 4747 kafka 159u  REG  8,16 536890888 2252895
> /mnt/kafka-logs/dispatch-0/00000000049396407038.kafka (deleted)
> java 4747 kafka 160u  REG  8,16 536923453 2252896
> /mnt/kafka-logs/dispatch-0/00000000049933297926.kafka (deleted)
> java 4747 kafka 161u  REG  8,16 536889637 2252897
> /mnt/kafka-logs/dispatch-0/00000000050470221379.kafka (deleted)
>
>
> When I restart the kafka process, it spends a nontrivial amount of time
> shutting down, doing a large amount of random reads on the disk before
> dying. I'm storing the logs on an Amazon EBS mount.
>
> Is there a way I can delete those files without needing restart?
>
> .Carlo
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message