lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomasz Kępski <tom...@kepski.pl>
Subject Huge load and long response times during search
Date Fri, 20 Nov 2009 20:15:08 GMT
Hi,

I'm using SOLR(1.4) to search among about 3,500,000 documents. After the 
server kernel was updated to 64bit system has started to suffer.
Our server has 8G of RAM and double Intel Core 2 DUO.
We used to have average loads around 2-2,5. It was not as good as it 
should but as long HTTP response times was acceptable we do not care to 
much ;-)

Since few days avg loads are usually around 6, sometimes goes even to 
20. PHP, Mysql and Postgresql based application is rather fine, but when 
tries to access SOLR it takes ages to load page. In top java process 
(Jetty) takes 200-250% of CPU, iotop shows that most of the disk 
operations are done by SOLR threads as well.

When we do shut down Jetty load goes down to 1,5 or even less than 1.

My index has ~12G below is a part of my solrconf.xml:

 <query>
    <maxBooleanClauses>1024</maxBooleanClauses>
    <filterCache
      class="solr.LRUCache"
      size="16384"
      initialSize="4096"
      autowarmCount="4096"/>
    <queryResultCache
      class="solr.LRUCache"
      size="16384"
      initialSize="4096"
      autowarmCount="1024"/>
    <documentCache
      class="solr.LRUCache"
      size="16384"
      initialSize="16384"
      autowarmCount="0"/>
    <enableLazyFieldLoading>true</enableLazyFieldLoading>
    <useFilterForSortedQuery>true</useFilterForSortedQuery>
    <queryResultWindowSize>40</queryResultWindowSize>
    <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
    <HashDocSet maxSize="3000" loadFactor="0.75"/>
    <listener event="newSearcher" class="solr.QuerySenderListener">
      <arr name="queries">
        <lst> <str name="q">solr</str> <str name="start">0</str>
<str 
name="rows">10</str> </lst>
        <lst> <str name="q">solr</str> <str name="sort">price</str>
<str 
name="start">0</str> <str name="rows">10</str> </lst>
        <lst> <str name="q">solr</str> <str 
name="sort">rekomendacja</str> <str name="start">0</str> <str 
name="rows">10</str> </lst>
        <lst><str name="q">static newSearcher warming query from 
solrconfig.xml</str></lst>
      </arr>
    </listener>
    <listener event="firstSearcher" class="solr.QuerySenderListener">
      <arr name="queries">
        <lst> <str name="q">fast_warm</str> <str name="start">0</str>

<str name="rows">10</str> </lst>
        <lst><str name="q">static firstSearcher warming query from 
solrconfig.xml</str></lst>
      </arr>
    </listener>
    <useColdSearcher>false</useColdSearcher>
</query>

  <requestHandler name="dismax" class="solr.SearchHandler" >
    <lst name="defaults">
     <str name="defType">dismax</str>
     <str name="echoParams">explicit</str>
     <float name="tie">0.01</float>
     <str name="qf">
        name^90.0 scategory^450.0 brand^90.0 text^0.01 description^30
     </str>
     <str name="pf">
     </str>
     <str name="bf">
     </str>
     <str name="fl">
        brand,description,id,name,price,score
     </str>
     <str name="mm">
        4&lt;100% 5&lt;90%
     </str>
     <int name="ps">100</int>
     <str name="q.alt">*:*</str>
    </lst>
  </requestHandler>

sample query parameters from log looks like this:

2009-11-20 21:07:15 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select 
params={spellcheck=true&wt=json&rows=20&json.nl=map&start=520&facet=true&spellcheck.collate=true&fl=id,name,description,preparation,url,shop_id&q=camera&qt=dismax&version=1.3&hl.fl=name,description,atributes,brand,url&facet.field=shop_id&facet.field=brand&hl.fragsize=200&spellcheck.count=5&hl.snippets=3&hl=true}

hits=3784 status=0 QTime=83
2009-11-20 21:07:15 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/spellCheckCompRH 
params={spellcheck=true&wt=json&rows=20&json.nl=map&start=520&facet=true&spellcheck.collate=true&fl=id,name,description,preparation,url,shop_id&q=camera&qt=dismax&version=1.3&hl.fl=name,description,atributes,brand,url&facet.field=shop_id&facet.field=brand&hl.fragsize=200&spellcheck.count=5&hl.snippets=3&hl=true}

hits=3784 status=0 QTime=16

And at last the question ;-)
How to speed up the search?
Which parameters should I check first to find out what is the bottleneck?

Sorry for verbose entry but I would like to give as clear point of view 
as possible

Thanks in advance,
Tom

Mime
View raw message