lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: growing segments issue in lucene
Date Thu, 29 Sep 2011 11:31:32 GMT
How about using MMapDirectory? The speed is similar to RAMDirectory and
often even faster when GC goes crazy.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: suman.holani [mailto:suman.holani@zapak.co.in]
> Sent: Thursday, September 29, 2011 7:46 AM
> To: java-user@lucene.apache.org
> Subject: RE: growing segments issue in lucene
> 
> Hi Ian,
> Thanks for reply.
> 
> Let me reframe the question
> 
> My aim is to search fast in ever changing indexes.
> So I decided onto use RAM directory index.but since it is not persistent
.I need
> to copy onto FSFirectory , which is nothing but the replica of it for
failure
> scenarios.
> 
> RAMDirectory is used for adding/deleting and searching.I am doing
> optimization on it and then copying .I am assuming that since RAMdirectory
is
> optimized so copy of it would also be optimized. But when I look at
FSDirectory
> it is showing me too many files.
> 
> I am not getting why the FSdirectory is growing Also , How can I check if
> proper optimsation is done in RAMdirectory . all is in memory or does it
stores
> segments somewhere in disk also.
> 
> > merge factor is10
> >
> > max merge docs is2147483647
> 
> My concerned is growing segments not getting merged on optimization ( as I
> can see in my fsdirectory )irrespective of merge factor.
> 
> 
> Regards,
> suman
> 
> 
> -----Original Message-----
> From: Ian Lea [mailto:ian.lea@gmail.com]
> Sent: Wednesday, September 28, 2011 8:55 PM
> To: java-user@lucene.apache.org
> Subject: Re: growing segments issue in lucene
> 
> Seems a strange way of doing things.  What do you gain by messing around
> with RAMDirectory?
> 
> Anyway, you appear to just add docs to the file based index, so surely it
will
> keep growing?  You could try optimizing the file based index but that is
unlikely
> to gain you a huge amount unless it holds deleted docs.
> 
> 
> --
> Ian.
> 
> 
> On Wed, Sep 28, 2011 at 11:32 AM, suman.holani <suman.holani@zapak.co.in>
> wrote:
> > Hi ,
> >
> >
> >
> > I am using lucene 3.0.3.
> >
> > It seems like either the lucene indexes are not getting optimized
> > properly as the files created in lucene index dir is growing with
> > every doc added
> >
> > Please let me know the way to restrict the files and segments created .
> >
> >
> >
> > merge factor is10
> >
> > max merge docs is2147483647
> >
> >
> >
> >
> >
> > I am using RAM index to add doc ,optimising it and then copying it to
> > FSDirectory.
> >
> >
> >
> > indexWriter.optimize();
> >
> >               indexWriter.commit();
> >
> >               Directory.copy(RamDir, FSDirectory.open(FILE_INDEX_DIR),
> > false);  //hoping that optimized indexes will b transferred since m
> > optimizing ramdir before copying
> >
> >
> >
> >
> >
> > Files created in FSDirectory as shown as .and these are increasing
> > with a index change.
> >
> > _0.cfs   _16.frq  _1e.fdt  _1k.prx  _1s.fnm  _1y.tis  _4.nrm  _c.fdx
>  _i.tii
> > _q.frq        segments_l  _u.tii
> >
> > _0.cfx   _16.nrm  _1e.fdx  _1k.tii  _1s.frq  _20.fdt  _4.prx  _c.fnm
>  _i.tis
> > _q.nrm        segments_m  _u.tis
> >
> > _10.fdt  _16.prx  _1e.fnm  _1k.tis  _1s.nrm  _20.fdx  _4.tii  _c.frq
>  _k.fdt
> > _q.prx        segments_n  _w.fdt
> >
> > _10.fdx  _16.tii  _1e.frq  _1m.fdt  _1s.prx  _20.fnm  _4.tis  _c.nrm
>  _k.fdx
> > _q.tii        segments_o  _w.fdx
> >
> > _10.fnm  _16.tis  _1e.nrm  _1m.fdx  _1s.tii  _20.frq  _6.fdt  _c.prx
>  _k.fnm
> > _q.tis        segments_p  _w.fnm
> >
> > _10.frq  _18.fdt  _1e.prx  _1m.fnm  _1s.tis  _20.nrm  _6.fdx  _c.tii
>  _k.frq
> > segments_10   segments_q  _w.frq
> >
> > _10.nrm  _18.fdx  _1e.tii  _1m.frq  _1u.fdt  _20.prx  _6.fnm  _c.tis
>  _k.nrm
> > segments_11   segments_r  _w.nrm
> >
> > _10.prx  _18.fnm  _1e.tis  _1m.nrm  _1u.fdx  _20.tii  _6.frq  _e.fdt
>  _k.prx
> > segments_12   segments_s  _w.prx
> >
> > _10.tii  _18.frq  _1g.fdt  _1m.prx  _1u.fnm  _20.tis  _6.nrm  _e.fdx
>  _k.tii
> > segments_13   segments_t  _w.tii
> >
> > _10.tis  _18.nrm  _1g.fdx  _1m.tii  _1u.frq  _22.fdt  _6.prx  _e.fnm
>  _k.tis
> > segments_2    segments_u  _w.tis
> >
> > _12.fdt  _18.prx  _1g.fnm  _1m.tis  _1u.nrm  _22.fdx  _6.tii  _e.frq
>  _m.fdt
> > segments_3    segments_v  _y.fdt
> >
> > _12.fdx  _18.tii  _1g.frq  _1o.fdt  _1u.prx  _22.fnm  _6.tis  _e.nrm
>  _m.fdx
> > segments_4    segments_w  _y.fdx
> >
> > _12.fnm  _18.tis  _1g.nrm  _1o.fdx  _1u.tii  _22.frq  _8.fdt  _e.prx
>  _m.fnm
> > segments_5    segments_x  _y.fnm
> >
> > _12.frq  _1a.fdt  _1g.prx  _1o.fnm  _1u.tis  _22.nrm  _8.fdx  _e.tii
>  _m.frq
> > segments_6    segments_y  _y.frq
> >
> > _12.nrm  _1a.fdx  _1g.tii  _1o.frq  _1w.fdt  _22.prx  _8.fnm  _e.tis
>  _m.nrm
> > segments_7    segments_z  _y.nrm
> >
> > _12.prx  _1a.fnm  _1g.tis  _1o.nrm  _1w.fdx  _22.tii  _8.frq  _g.fdt
>  _m.prx
> > segments_8    _s.fdt      _y.prx
> >
> > _12.tii  _1a.frq  _1i.fdt  _1o.prx  _1w.fnm  _22.tis  _8.nrm  _g.fdx
>  _m.tii
> > segments_9    _s.fdx      _y.tii
> >
> > _12.tis  _1a.nrm  _1i.fdx  _1o.tii  _1w.frq  _2.fdt   _8.prx  _g.fnm
>  _m.tis
> > segments_a    _s.fnm      _y.tis
> >
> > _14.fdt  _1a.prx  _1i.fnm  _1o.tis  _1w.nrm  _2.fdx   _8.tii  _g.frq
>  _o.fdt
> > segments_b    _s.frq
> >
> > _14.fdx  _1a.tii  _1i.frq  _1q.fdt  _1w.prx  _2.fnm   _8.tis  _g.nrm
>  _o.fdx
> > segments_c    _s.nrm
> >
> > _14.fnm  _1a.tis  _1i.nrm  _1q.fdx  _1w.tii  _2.frq   _a.fdt  _g.prx
>  _o.fnm
> > segments_d    _s.prx
> >
> > _14.frq  _1c.fdt  _1i.prx  _1q.fnm  _1w.tis  _2.nrm   _a.fdx  _g.tii
>  _o.frq
> > segments_e    _s.tii
> >
> > _14.nrm  _1c.fdx  _1i.tii  _1q.frq  _1y.fdt  _2.prx   _a.fnm  _g.tis
>  _o.nrm
> > segments_f    _s.tis
> >
> > _14.prx  _1c.fnm  _1i.tis  _1q.nrm  _1y.fdx  _2.tii   _a.frq  _i.fdt
>  _o.prx
> > segments_g    _u.fdt
> >
> > _14.tii  _1c.frq  _1k.fdt  _1q.prx  _1y.fnm  _2.tis   _a.nrm  _i.fdx
>  _o.tii
> > segments.gen  _u.fdx
> >
> > _14.tis  _1c.nrm  _1k.fdx  _1q.tii  _1y.frq  _4.fdt   _a.prx  _i.fnm
>  _o.tis
> > segments_h    _u.fnm
> >
> > _16.fdt  _1c.prx  _1k.fnm  _1q.tis  _1y.nrm  _4.fdx   _a.tii  _i.frq
>  _q.fdt
> > segments_i    _u.frq
> >
> > _16.fdx  _1c.tii  _1k.frq  _1s.fdt  _1y.prx  _4.fnm   _a.tis  _i.nrm
>  _q.fdx
> > segments_j    _u.nrm
> >
> > _16.fnm  _1c.tis  _1k.nrm  _1s.fdx  _1y.tii  _4.frq   _c.fdt  _i.prx
>  _q.fnm
> > segments_k    _u.prx
> >
> >
> >
> > Regards
> >
> > suman
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message