lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fuad Efendi" <f...@efendi.ca>
Subject RE: Lucene FieldCache memory requirements
Date Tue, 03 Nov 2009 00:37:26 GMT

Simple field (10 different values: Canada, USA, UK, ...), 64-bit JVM... no
difference between maxdoc and maxdoc + 1 for such estimate... difference is
between 0.4Gb and 1.2Gb...


So, let's vote ;)

A. [maxdoc] x [8 bytes ~ pointer to String object]

B. [maxdoc] x [8 bytes ~ pointer to Document object]

C. [maxdoc] x [4 bytes ~ (int) Lucene Document ID] 
- same as [String1_Document_Count + ... + String10_Document_Count] x [4
bytes ~ DocumentID]

D. [maxdoc] x [4 bytes + 8 bytes ~ my initial naive thinking...]


Please confirm that it is Pointer to Object and not Lucene Document ID... I
hope it is (int) Document ID...





> -----Original Message-----
> From: Mark Miller [mailto:markrmiller@gmail.com]
> Sent: November-02-09 6:52 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Lucene FieldCache memory requirements
> 
> It also briefly requires more memory than just that - it allocates an
> array the size of maxdoc+1 to hold the unique terms - and then sizes down.
> 
> Possibly we can use the getUnuiqeTermCount method in the flexible
> indexing branch to get rid of that - which is why I was thinking it
> might be a good idea to drop the unsupported exception in that method
> for things like multi reader and just do the work to get the right
> number (currently there is a comment that the user should do that work
> if necessary, making the call unreliable for this).
> 
> Fuad Efendi wrote:
> > Thank you very much Mike,
> >
> > I found it:
> > org.apache.solr.request.SimpleFacets
> > ...
> >         // TODO: future logic could use filters instead of the
fieldcache if
> >         // the number of terms in the field is small enough.
> >         counts = getFieldCacheCounts(searcher, base, field,
offset,limit,
> > mincount, missing, sort, prefix);
> > ...
> >     FieldCache.StringIndex si =
> > FieldCache.DEFAULT.getStringIndex(searcher.getReader(), fieldName);
> >     final String[] terms = si.lookup;
> >     final int[] termNum = si.order;
> > ...
> >
> >
> > So that 64-bit requires more memory :)
> >
> >
> > Mike, am I right here?
> > [(8 bytes pointer) + (4 bytes DocID)] x [Number of Documents (100mlns)]
> > (64-bit JVM)
> > 1.2Gb RAM for this...
> >
> > Or, may be I am wrong:
> >
> >> For Lucene directly, simple strings would consume an pointer (4 or 8
> >> bytes depending on whether your JRE is 64bit) per doc, and the string
> >> index would consume an int (4 bytes) per doc.
> >>
> >
> > [8 bytes (64bit)] x [number of documents (100mlns)]?
> > 0.8Gb
> >
> > Kind of Map between String and DocSet, saving 4 bytes... "Key" is
String,
> > and "Value" is array of 64-bit pointers to Document. Why 64-bit (for
64-bit
> > JVM)? I always thought it is (int) documentId...
> >
> > Am I right?
> >
> >
> > Thanks for pointing to http://issues.apache.org/jira/browse/LUCENE-1990!
> >
> >
> >>> Note that for your use case, this is exceptionally wasteful.
> >>>
> > This is probably very common case... I think it should be confirmed by
> > Lucene developers too... FieldCache is warmed anyway, even when we don't
use
> > SOLR...
> >
> >
> > -Fuad
> >
> >
> >
> >
> >
> >
> >
> >
> >> -----Original Message-----
> >> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> >> Sent: November-02-09 6:00 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: Re: Lucene FieldCache memory requirements
> >>
> >> OK I think someone who knows how Solr uses the fieldCache for this
> >> type of field will have to pipe up.
> >>
> >> For Lucene directly, simple strings would consume an pointer (4 or 8
> >> bytes depending on whether your JRE is 64bit) per doc, and the string
> >> index would consume an int (4 bytes) per doc.  (Each also consume
> >> negligible (for your case) memory to hold the actual string values).
> >>
> >> Note that for your use case, this is exceptionally wasteful.  If
> >> Lucene had simple bit-packed ints (I've opened LUCENE-1990 for this)
> >> then it'd take much fewer bits to reference the values, since you have
> >> only 10 unique string values.
> >>
> >> Mike
> >>
> >> On Mon, Nov 2, 2009 at 3:57 PM, Fuad Efendi <fuad@efendi.ca> wrote:
> >>
> >>> I am not using Lucene API directly; I am using SOLR which uses Lucene
> >>> FieldCache for faceting on non-tokenized fields...
> >>> I think this cache will be lazily loaded, until user executes sorted
(by
> >>> this field) SOLR query for all documents *:* - in this case it will be
> >>>
> > fully
> >
> >>> populated...
> >>>
> >>>
> >>>
> >>>> Subject: Re: Lucene FieldCache memory requirements
> >>>>
> >>>> Which FieldCache API are you using?  getStrings?  or getStringIndex
> >>>> (which is used, under the hood, if you sort by this field).
> >>>>
> >>>> Mike
> >>>>
> >>>> On Mon, Nov 2, 2009 at 2:27 PM, Fuad Efendi <fuad@efendi.ca> wrote:
> >>>>
> >>>>> Any thoughts regarding the subject? I hope FieldCache doesn't use
> >>>>>
> > more
> >
> >>> than
> >>>
> >>>>> 6 bytes per document-field instance... I am too lazy to research
> >>>>>
> > Lucene
> >
> >>>>> source code, I hope someone can provide exact answer... Thanks
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Subject: Lucene FieldCache memory requirements
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>
> >>>>>> Can anyone confirm Lucene FieldCache memory requirements? I
have
100
> >>>>>> millions docs with non-tokenized field "country" (10 different
> >>>>>>
> >>> countries);
> >>>
> >>>>> I
> >>>>>
> >>>>>> expect it requires array of ("int", "long"), size of array
> >>>>>>
> > 100,000,000,
> >
> >>>>>> without any impact of "country" field length;
> >>>>>>
> >>>>>> it requires 600,000,000 bytes: "int" is pointer to document
(Lucene
> >>>>>>
> >>>>> document
> >>>>>
> >>>>>> ID),  and "long" is pointer to String value...
> >>>>>>
> >>>>>> Am I right, is it 600Mb just for this "country" (indexed,
> >>>>>>
> >>> non-tokenized,
> >>>
> >>>>>> non-boolean) field and 100 millions docs? I need to calculate
exact
> >>>>>>
> >>>>> minimum RAM
> >>>>>
> >>>>>> requirements...
> >>>>>>
> >>>>>> I believe it shouldn't depend on cardinality (distribution)
of
> >>>>>>
> > field...
> >
> >>>>>> Thanks,
> >>>>>> Fuad
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
> >
> >
> 
> 
> --
> - Mark
> 
> http://www.lucidimagination.com
> 
> 
- Fuad

http://www.linkedin.com/in/liferay



Mime
View raw message