kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: broker not release segment file after deleting expired segments?
Date Wed, 08 Aug 2012 22:55:31 GMT
Hmm, this seems like an issue with du.

Thanks

Jun

On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <xiaoyu.wang@gmail.com> wrote:

> Here is a sample. The highlighted lines are those showing different sizes
> in du and ls
>
> *du output
> *
> 513M    ./00000001197773741184.kafka
> 1.0G    ./00000001200458096891.kafka
> 1.0G    ./00000001200994968784.kafka
> 1.0G    ./00000001201531839884.kafka
> 1.0G    ./00000001202068711094.kafka
> 513M    ./00000001011479389046.kafka
>
> *ls output
> *
> -rw-r--r--  1 kafka kafka 536871053 Aug  8 13:01 00000001197773741184.kafka
> -rw-r--r--  1 kafka kafka 536871893 Aug  8 13:36 00000001200458096891.kafka
> -rw-r--r--  1 kafka kafka 536871100 Aug  8 13:45 00000001200994968784.kafka
> -rw-r--r--  1 kafka kafka 536871210 Aug  8 13:49 00000001201531839884.kafka
> -rw-r--r--  1 kafka kafka 536871063 Aug  8 14:00 00000001202068711094.kafka
> -rw-r--r--  1 kafka kafka 536871139 Aug  6 07:02 00000001011479389046.kafka
>
> On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:
>
> > The mailing list does not accept attachments - you will need to either
> > paste here or externally, say on gist.
> >
> > Thanks,
> >
> > Joel
> >
> > On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <xiaoyu.wang@gmail.com>
> wrote:
> >
> > > Neha,
> > >
> > > I have attached the du and ls output for one of the partitions. Looks
> > like
> > > for some segments, du is showing 1GB while ls shows 513MB. Let me know
> if
> > > you need additional info.
> > >
> > > Thanks,
> > >
> > > -Xiaoyu
> > >
> > >
> > >
> > > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > >wrote:
> > >
> > >> Xiaoyu,
> > >>
> > >> Please could you paste the log statements that specify which files are
> > >> deleted by the broker and also the detailed output by du on the Kafka
> > >> log directory ?
> > >> If the broker says it has garbage collected some log segments, space
> > >> should free up.
> > >>
> > >> Thanks,
> > >> Neha
> > >>
> > >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <xiaoyu.wang@gmail.com>
> > >> wrote:
> > >> > Hi all,
> > >> >
> > >> > I am running kafka-0.7.0 and found that even though expired segments
> > are
> > >> > deleted by broker, the disk space is not released. So, du shows
> that a
> > >> > partition is 450 GB, but the total size of the files in the
> partition
> > is
> > >> > only about 250GB. Is this a bug? We are using raid10.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > -Xiaoyu
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message