lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Khludnev <mkhlud...@griddynamics.com>
Subject Re:
Date Sat, 17 May 2014 12:20:55 GMT
For Sure. Lucene's explain is really expensive and is not purposed for
production use, but only for rare troubleshooting. As a mitigation measure
you can scroll result set by small portions more efficient like Hoss
recently explained at SearchHub. In such kind of problems, usually it's
possible to create sort of specialized custom collectors doing something
particular.

Have a god day!


On Sat, May 17, 2014 at 3:01 AM, Tom Burton-West <tburtonw@umich.edu> wrote:

> Hello all,
>
>
> I'm trying to get relevance scoring information for each of 1,000 docs
> returned for each of 250 queries.    If I run the query (appended below)
> without debugQuery=on, I have no problem with getting all the results
> with under 4GB of memory use.  If I add the parameter &debugQuery=on,
> memory use goes up continuously and after about 20 queries (with 1,000
> results each), memory use reaches about 29.1 GB and the garbage collector
> gives up:
>
> " org.apache.solr.common.SolrException; null:java.lang.RuntimeException:
> java.lang.OutOfMemoryError: GC overhead limit exceeded"
>
> I've attached a jmap -histo, exgerpt below.
>
> Is this a known issue with debugQuery?
>
> Tom
> ----
> query:
>
>
> q=Abraham+Lincoln&fl=id,score&indent=on&wt=json&start=0&rows=1000&version=2.2&
> debugQuery=on
>
> without debugQuery=on:
>
>
> q=Abraham+Lincoln&fl=id,score&indent=on&wt=json&start=0&rows=1000&version=2.2
>
> num       #instances    #bytes  Class description
> --------------------------------------------------------------------------
> 1:              585,559 10,292,067,456  byte[]
> 2:              743,639 18,874,349,592  char[]
> 3:              53,821  91,936,328      long[]
> 4:              70,430  69,234,400      int[]
> 5:              51,348  27,111,744
>  org.apache.lucene.util.fst.FST$Arc[]
> 6:              286,357 20,617,704      org.apache.lucene.util.fst.FST$Arc
> 7:              715,364 17,168,736      java.lang.String
> 8:              79,561  12,547,792      * ConstMethodKlass
> 9:              18,909  11,404,696      short[]
> 10:             345,854 11,067,328      java.util.HashMap$Entry
> 11:             8,823   10,351,024      * ConstantPoolKlass
> 12:             79,561  10,193,328      * MethodKlass
> 13:             228,587 9,143,480
> org.apache.lucene.document.FieldType
> 14:             228,584 9,143,360       org.apache.lucene.document.Field
> 15:             368,423 8,842,152       org.apache.lucene.util.BytesRef
> 16:             210,342 8,413,680       java.util.TreeMap$Entry
> 17:             81,576  8,204,648       java.util.HashMap$Entry[]
> 18:             107,921 7,770,312       org.apache.lucene.util.fst.FST$Arc
> 19:             13,020  6,874,560
> org.apache.lucene.util.fst.FST$Arc[]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
<mkhludnev@griddynamics.com>

Mime
View raw message