lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gimantha Bandara <giman...@wso2.com>
Subject Re: How to merge several Taxonomy indexes
Date Thu, 02 Apr 2015 11:46:19 GMT
Hi Christoph and Shai,

Thanks for the quick response!.
Indices are stored in a relational database ( using a custom Directory
implementation ). The Problem comes since the indices are sharded (both
taxonomy indices and normal doc indices), when a user wants to drilldown, I
have to merge all the indices. For that I used mergeUtils (which
worksperfect). For now I am using RAMDirectory as the merged indices.
Anyway The indices can grow to a bigger size as time goes. MMapDirectory
again uses memory right? Can It deal with possible out of memory issue?

I am thinking of using the same Database to store the merged indices. But
the problem is the original sharded indices can be updated, when new
entries come in. So the merged final indices also needs to be updated
accordingly.

On Thu, Apr 2, 2015 at 4:55 PM, Shai Erera <serera@gmail.com> wrote:

> In some cases, MMapDirectory offers even better performance, since the JVM
> doesn't need to manage that RAM when it's doing GC.
>
> Also, using only RAMDirectory is not safe in that if the JVM crashes, your
> index is lost.
>
> On Thu, Apr 2, 2015 at 12:54 PM, Christoph Kaser <lucene_list@iconparc.de>
> wrote:
>
> > Hi Gimantha,
> >
> > why do you use a RAMDirectory? If your merged index fits into RAM
> > completely, a MMapDirectory should offer almost the same performance. And
> > if not, it is definitely the better choice.
> >
> > Regards
> > Christoph
> >
> >
> > Am 02.04.2015 um 12:38 schrieb Gimantha Bandara:
> >
> >> Hi All,
> >>
> >> I have successfully setup a merged indices and drilldown and usual
> search
> >> operations work perfect.
> >> But, I have a side question. If I selected RAMDirectory as the
> destination
> >> Indices in merging, probably the jvm can go out of memory if the merged
> >> indices are too big. Is there a way I can handle this issue?
> >>
> >> On Tue, Mar 24, 2015 at 12:18 PM, Gimantha Bandara <gimantha@wso2.com>
> >> wrote:
> >>
> >>  Hi Christoph,
> >>>
> >>> My mistake. :) It does the exactly what i need. figured it out later..
> >>> Thanks a lot!
> >>>
> >>> On Tue, Mar 24, 2015 at 3:14 AM, Gimantha Bandara <gimantha@wso2.com>
> >>> wrote:
> >>>
> >>>  Hi Christoph,
> >>>>
> >>>> I think TaxonomyMergeUtils is to merge a taxonomy directory and an
> index
> >>>> together (Correct me if I am wrong). Can it be used to merge several
> >>>> taxonomyDirectories together and create one taxonomy index?
> >>>>
> >>>> On Mon, Mar 23, 2015 at 9:19 PM, Christoph Kaser <
> >>>> lucene_list@iconparc.de
> >>>>
> >>>>> wrote:
> >>>>> Hi Gimantha,
> >>>>>
> >>>>> have a look at the class org.apache.lucene.facet.
> >>>>> taxonomy.TaxonomyMergeUtils,
> >>>>> which does exactly what you need.
> >>>>>
> >>>>> Best regards,
> >>>>> Christoph
> >>>>>
> >>>>> Am 23.03.2015 um 15:44 schrieb Gimantha Bandara:
> >>>>>
> >>>>>  Hi all,
> >>>>>>
> >>>>>> Can anyone point me how to merge several taxonomy indexes? My
> >>>>>> requirement
> >>>>>> is as follows. I have  several taxonomy indexes and normal document
> >>>>>> indexes. I want to merge taxonomy indexes together and other
> document
> >>>>>> indexes together and perform search on them. One part I have
figured
> >>>>>> out.
> >>>>>> It is easy. To Merge document indexes, all I have to do is create
a
> >>>>>> MultiReader and pass it to IndexSearcher. But I am stuck at
merging
> >>>>>> the
> >>>>>> taxonomy indexes. Is there a way to merge taxonomy indexes?
> >>>>>>
> >>>>>>
> >>>>>>  --
> >>>>> Dipl.-Inf. Christoph Kaser
> >>>>>
> >>>>> IconParc GmbH
> >>>>> Sophienstrasse 1
> >>>>> 80333 München
> >>>>>
> >>>>> www.iconparc.de
> >>>>>
> >>>>> Tel +49 -89- 15 90 06 - 21
> >>>>> Fax +49 -89- 15 90 06 - 49
> >>>>>
> >>>>> Geschäftsleitung: Dipl.-Ing. Roland Brückner, Dipl.-Inf. Sven
> Angerer.
> >>>>> HRB
> >>>>> 121830, Amtsgericht München
> >>>>>
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>>
> >>>>>
> >>>>>  --
> >>>> Gimantha Bandara
> >>>> Software Engineer
> >>>> WSO2. Inc : http://wso2.com
> >>>> Mobile : +94714961919
> >>>>
> >>>>  --
> >>> Gimantha Bandara
> >>> Software Engineer
> >>> WSO2. Inc : http://wso2.com
> >>> Mobile : +94714961919
> >>>
> >>>
> >>
> >
> > --
> > Dipl.-Inf. Christoph Kaser
> >
> > IconParc GmbH
> > Sophienstrasse 1
> > 80333 München
> >
> > www.iconparc.de
> >
> > Tel +49 -89- 15 90 06 - 21
> > Fax +49 -89- 15 90 06 - 49
> >
> > Geschäftsleitung: Dipl.-Ing. Roland Brückner, Dipl.-Inf. Sven Angerer.
> HRB
> > 121830, Amtsgericht München
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>



-- 
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : +94714961919

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message