lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gimantha Bandara <giman...@wso2.com>
Subject Re: How to read multiple indices in parallel.
Date Tue, 07 Apr 2015 14:50:16 GMT
That was really helpful. Thanks a lot Terry!

On Tue, Apr 7, 2015 at 8:17 PM, Terry Smith <shebiki@gmail.com> wrote:

> Gimantha,
>
> Search will run in parallel even across indices.
>
> This happens because IndexSearcher searches by LeafReader and it doesn't
> matter where those LeafReaders come from (DirectoryReader or MultiReader)
> they are all treated equally.
>
> Example:
>
> DirectoryReader(A):
>     LeafReader(B), LeafReader(C)
>
> DirectoryReader(D):
>     LeafReader(E), LeafReader(F)
>
> Searching over A would use leaves B, C.
> Searching over B would use leaves E, F.
> Searching over a MultiReader on (A, B) would use leaves B, C, E, F.
>
> This runs serially by default but can run in parallel if you provide an
> ExecutorService to the IndexSearcher and use a compatible search() method
> on it.
>
> --Terry
>
>
> On Tue, Apr 7, 2015 at 10:27 AM, Gimantha Bandara <gimantha@wso2.com>
> wrote:
>
> > Hi Terry,
> >
> > I have multiple indices in separate locations. If I used multireader and
> > used an executorservice with the indexSearcher It will go thru the
> segments
> > in parallel and search right? But still searching between different
> indices
> > will happen sequentially..Isnt it?
> >
> > On Tue, Apr 7, 2015 at 7:15 PM, Terry Smith <shebiki@gmail.com> wrote:
> >
> > > Gimantha,
> > >
> > > With Lucene 5.0 you can pass in an ExecutorService to the constructor
> of
> > > your IndexSearcher and it will search the segments in parallel if you
> use
> > > one of the IndexSearcher.search() methods that returns a TopDocs (and
> > don't
> > > supply your own Collector).
> > >
> > > The not-yet-released Lucene 5.1 includes some changes (LUCENE-6294
> > > <https://issues.apache.org/jira/browse/LUCENE-6294>) that enable
> better
> > > parallel query support.
> > >
> > > --Terry
> > >
> > >
> > > On Tue, Apr 7, 2015 at 8:09 AM, Gimantha Bandara <gimantha@wso2.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > As I can see the Multireader is reading the multiple indices
> > sequentially
> > > > (correct me if I am wrong). So using a IndexSearcher on a multireader
> > > will
> > > > also perform sequential searches right? Is there a lucene-built-in
> > class
> > > to
> > > > search several indices parallely?
> > > >
> > > > --
> > > > Gimantha Bandara
> > > > Software Engineer
> > > > WSO2. Inc : http://wso2.com
> > > > Mobile : +94714961919
> > > >
> > >
> >
> >
> >
> > --
> > Gimantha Bandara
> > Software Engineer
> > WSO2. Inc : http://wso2.com
> > Mobile : +94714961919
> >
>



-- 
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : +94714961919

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message