kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: broker not release segment file after deleting expired segments?
Date Thu, 09 Aug 2012 17:53:08 GMT
There is no preallocation in Kafka. I do remember seeing incorrect results
from du before.

Thanks,

Jun

On Thu, Aug 9, 2012 at 10:29 AM, xiaoyu wang <xiaoyu.wang@gmail.com> wrote:

> Does kafka pre-allocate 1GB for each segment? It shows that these segment
> files are all open and used by kafka broker.
>
> On Wed, Aug 8, 2012 at 4:14 PM, xiaoyu wang <xiaoyu.wang@gmail.com> wrote:
>
> > Jun,
> >
> > this does not seem to be a du problem because after restarting brokers,
> du
> > and ls show the same file size.
> >
> >
> > On Wed, Aug 8, 2012 at 3:55 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> >> Hmm, this seems like an issue with du.
> >>
> >> Thanks
> >>
> >> Jun
> >>
> >> On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <xiaoyu.wang@gmail.com>
> >> wrote:
> >>
> >> > Here is a sample. The highlighted lines are those showing different
> >> sizes
> >> > in du and ls
> >> >
> >> > *du output
> >> > *
> >> > 513M    ./00000001197773741184.kafka
> >> > 1.0G    ./00000001200458096891.kafka
> >> > 1.0G    ./00000001200994968784.kafka
> >> > 1.0G    ./00000001201531839884.kafka
> >> > 1.0G    ./00000001202068711094.kafka
> >> > 513M    ./00000001011479389046.kafka
> >> >
> >> > *ls output
> >> > *
> >> > -rw-r--r--  1 kafka kafka 536871053 Aug  8 13:01
> >> 00000001197773741184.kafka
> >> > -rw-r--r--  1 kafka kafka 536871893 Aug  8 13:36
> >> 00000001200458096891.kafka
> >> > -rw-r--r--  1 kafka kafka 536871100 Aug  8 13:45
> >> 00000001200994968784.kafka
> >> > -rw-r--r--  1 kafka kafka 536871210 Aug  8 13:49
> >> 00000001201531839884.kafka
> >> > -rw-r--r--  1 kafka kafka 536871063 Aug  8 14:00
> >> 00000001202068711094.kafka
> >> > -rw-r--r--  1 kafka kafka 536871139 Aug  6 07:02
> >> 00000001011479389046.kafka
> >> >
> >> > On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <jjkoshy.w@gmail.com>
> wrote:
> >> >
> >> > > The mailing list does not accept attachments - you will need to
> either
> >> > > paste here or externally, say on gist.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Joel
> >> > >
> >> > > On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <xiaoyu.wang@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Neha,
> >> > > >
> >> > > > I have attached the du and ls output for one of the partitions.
> >> Looks
> >> > > like
> >> > > > for some segments, du is showing 1GB while ls shows 513MB. Let
me
> >> know
> >> > if
> >> > > > you need additional info.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > -Xiaoyu
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <
> >> > neha.narkhede@gmail.com
> >> > > >wrote:
> >> > > >
> >> > > >> Xiaoyu,
> >> > > >>
> >> > > >> Please could you paste the log statements that specify which
> files
> >> are
> >> > > >> deleted by the broker and also the detailed output by du
on the
> >> Kafka
> >> > > >> log directory ?
> >> > > >> If the broker says it has garbage collected some log segments,
> >> space
> >> > > >> should free up.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> Neha
> >> > > >>
> >> > > >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <
> >> xiaoyu.wang@gmail.com>
> >> > > >> wrote:
> >> > > >> > Hi all,
> >> > > >> >
> >> > > >> > I am running kafka-0.7.0 and found that even though
expired
> >> segments
> >> > > are
> >> > > >> > deleted by broker, the disk space is not released. So,
du shows
> >> > that a
> >> > > >> > partition is 450 GB, but the total size of the files
in the
> >> > partition
> >> > > is
> >> > > >> > only about 250GB. Is this a bug? We are using raid10.
> >> > > >> >
> >> > > >> > Thanks,
> >> > > >> >
> >> > > >> > -Xiaoyu
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message