lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: RE: RE: Would docvalues be loaded into jvm?
Date Thu, 15 Jun 2017 11:44:50 GMT
Yes.

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: wangqinghuan [mailto:1095193290@qq.com]
> Sent: Thursday, June 15, 2017 12:21 PM
> To: general@lucene.apache.org
> Subject: Re: RE: RE: Would docvalues be loaded into jvm?
> 
> Does "hotspot" reffers to java virtual machine?
> 
> 
> 
> ---Original---
> From: "Uwe Schindler [via
> Lucene]"<ml+s472066n4340678h46@n3.nabble.com>
> Date: 2017/6/15 17:03:46
> To: "wangqinghuan"<1095193290@qq.com>;
> Subject: RE: RE: Would docvalues be loaded into jvm?
> 
> 
>  Hi,
> 
> There is no design document about that. Lucene uses MMAP for all index
> files since a long time ago. DocValues is just another implementation.
> Basically it uses IndexInput's methods to access the underlying data, which is
> memory mapped if you are on 64 bit platforms. For DocValues there are also
> positional reads available. There is not much stuff specifically for docvalues,
> it is just a file format that supports column based access with positional
> reads. The mmap implementation is separated from this and a bit lower in
> the I/O layer of Lucene. Sorting is just a use case of DocValues, but it does
> not sort directly on the mmapped files, there are several abstractions
> inbetween (which are of course removed by the Hotspot optimizer).
> 
> Some information (a bit older, but still valid) is here:
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> 
> Uwe
> 
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: [hidden email]
> 
> > -----Original Message-----
> > From: wangqinghuan [mailto:[hidden email]]
> > Sent: Thursday, June 15, 2017 10:41 AM
> > To: [hidden email]
> > Subject: Re: RE: Would docvalues be loaded into jvm?
> >
> > hi
> > Is there any  design document on this aspect (sorting algorithm off mmap)?
> >
> >
> >
> > ---Original---
> > From: "Uwe Schindler [via
> > Lucene]"<[hidden email]>
> > Date: 2017/6/15 14:39:30
> > To: "wangqinghuan"<[hidden email]>;
> > Subject: RE: Would docvalues be loaded into jvm?
> >
> >
> >  Hi
> >
> > It works directly off  the mmapped files. It is not fully loaded into heap,
> only
> > some small control structures are allocated on heap. During sorting the
> > TopDocsCollector uses the memory mapped structures to uncompress and
> > lookup the sort values.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > http://www.thetaphi.de
> > eMail: [hidden email]
> >
> > > -----Original Message-----
> > > From: wangqinghuan [mailto:[hidden email]]
> > > Sent: Thursday, June 15, 2017 4:36 AM
> > > To: [hidden email]
> > > Subject: Would docvalues be loaded into jvm?
> > >
> > > hi
> > > I know that data is written into disk with the style of column-store if I
> > > enable doc-values for certain field.
> > > But I don't understand why sorting with docvalues doesn't increase the
> > load
> > > of jvm. whatever sorting algorithm , data would be loaded into  jvm to
> sort.
> > > This should be a high load for jvm when I sort all index  , but  no change
> > > for jvm in fact.  How does lucene sort with docvalues ? Can sort algorithm
> > > work directly based on the file (Mmap) ?
> > >
> > >
> > >
> > > --
> > > View this message in context:
> http://lucene.472066.n3.nabble.com/Would-
> > > docvalues-be-loaded-into-jvm-tp4340644.html
> > > Sent from the Lucene - General mailing list archive at Nabble.com.
> >
> >
> >
> >
> >   If you reply to this email, your message will be added to the discussion
> > below:
> >  http://lucene.472066.n3.nabble.com/Would-docvalues-be-loaded-into-
> jvm-
> > tp4340644p4340659.html
> >   To unsubscribe from Would docvalues be loaded into jvm?, click here.
> >  NAML
> >
> >
> >
> > --
> > View this message in context: http://lucene.472066.n3.nabble.com/Would-
> > docvalues-be-loaded-into-jvm-tp4340644p4340667.html
> > Sent from the Lucene - General mailing list archive at Nabble.com.
> 
> 
> 
> 
>   If you reply to this email, your message will be added to the discussion
> below:
>  http://lucene.472066.n3.nabble.com/Would-docvalues-be-loaded-into-jvm-
> tp4340644p4340678.html
>   To unsubscribe from Would docvalues be loaded into jvm?, click here.
>  NAML
> 
> 
> 
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Would-
> docvalues-be-loaded-into-jvm-tp4340644p4340689.html
> Sent from the Lucene - General mailing list archive at Nabble.com.


Mime
View raw message