lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Andersson <dan...@polkabrothers.com>
Subject Help optimizing
Date Sat, 03 May 2008 17:06:10 GMT
Hi (again) people

We've now invested in a server with 8 GB of RAM after too many  
OutOfMemory-errors.

Our database/index is 3.5 GB and contains 4,352,471 documents. Most  
documents are less than 1 kb. When performing a search, the results  
vary between 1.5 seconds up to 60 seconds.

I don't have a big problem with 1.5 seconds (even though below 1 would  
be nice), but 60 seconds it just.. well, scary.

How do I optimize Solr to better use all the RAM? I'm using java6,  
64bit version, and start Solr using:
java -Xmx7500M -Xms4096M -jar start.jar

But according to top it only seems to be using 7.7% of the memory  
(around 600 MB).

Most queries are for make_id + model_id or city + state and almost all  
of the queries are ordered by datetime_found (newest -> oldest).

from schema.xml:
    <field name="ad_id" type="sint" indexed="true" stored="true"  
required="true" />
    <field name="make_id" type="sint" indexed="true"  
multiValued="false" />
    <field name="model_id" type="sint" indexed="true"  
multiValued="false" />
    <field name="model_group_id" type="sint" indexed="true"  
multiValues="true" />
    <field name="zip_code" type="sint" indexed="true"  
multiValued="false" />
    <field name="year" type="sint" indexed="true" multiValued="false" />
    <field name="mileage" type="sint" indexed="true"  
multiValued="false" />
    <field name="cylinders" type="sint" indexed="true"  
multiValued="false" />
    <field name="doors" type="sint" indexed="true"  
multiValued="false" />
    <field name="price" type="sfloat" indexed="true"  
multiValued="false" />
    <field name="headline" type="text" indexed="true"  
multiValued="false" />
    <field name="description" type="text" indexed="true"  
multiValued="false" />
    <field name="condition" type="text" indexed="true"  
multiValued="false" />
    <field name="sell_type" type="text" indexed="true"  
multiValued="false" />
    <field name="city" type="text" indexed="true" multiValued="false" />
    <field name="state" type="text" indexed="true"  
multiValued="false" />
    <field name="alt_location" type="text" indexed="true"  
multiValued="false" />
    <field name="address" type="text" indexed="true"  
multiValued="false" />
    <field name="image_exists" type="boolean" indexed="true"  
multiValued="false" />
    <field name="vehicle_type" type="text" indexed="true"  
multiValued="false" />
    <field name="datetime_found" type="date" indexed="true"  
multiValued="false" />
    <field name="transmission" type="text" indexed="true"  
multiValued="false" />
    <field name="color" type="text" indexed="true"  
multiValued="false" />
    <field name="interior" type="text" indexed="true"  
multiValued="false" />
    <field name="private_seller" type="boolean" indexed="true"  
multiValued="false" />
    <field name="fuel" type="text" indexed="true" multiValued="false" />
    <field name="ac" type="boolean" indexed="true"  
multiValued="false" />
    <field name="cruise_control" type="boolean" indexed="true"  
multiValued="false" />
    <field name="lat" type="sdouble" indexed="true" stored="true" />
    <field name="lng" type="sdouble" indexed="true" stored="true" />
    <dynamicField name="_local*" type="sdouble" indexed="true"  
stored="true" multiValued="false" />

    <!-- catchall field, containing all other searchable text fields  
(implemented
         via copyField further on in this schema  -->
    <field name="text" type="text" indexed="true" stored="false"  
multiValued="true"/>

    <copyField source="headline" dest="text"/>
    <copyField source="description" dest="text"/>
    <copyField source="color" dest="text"/>


from solrconfig.xml:
     <filterCache
       class="solr.LRUCache"
       size="4096"
       initialSize="4096"
       autowarmCount="2048"/>

     <queryResultCache
       class="solr.LRUCache"
       size="4096"
       initialSize="4096"
       autowarmCount="2048"/>

     <documentCache
       class="solr.LRUCache"
       size="4096"
       initialSize="4096"
       autowarmCount="0"/>

     <queryResultWindowSize>50</queryResultWindowSize>

     <queryResultMaxDocsCached>1000</queryResultMaxDocsCached>

     <HashDocSet maxSize="2000" loadFactor="0.75"/>
     <listener event="newSearcher" class="solr.QuerySenderListener">
       <arr name="queries">
         <lst> <str name="q">datetime_found</str> <str name="start">0</

str> <str name="rows">10</str> </lst>
       </arr>
     </listener>

     <listener event="firstSearcher" class="solr.QuerySenderListener">
       <arr name="queries">
         <lst> <str name="q">datetime_found</str> <str name="start">0</

str> <str name="rows">10</str> </lst>
       </arr>
     </listener>

     <useColdSearcher>true</useColdSearcher>
     <maxWarmingSearchers>6</maxWarmingSearchers>

Any help much appreciated!

Kind regards,
Daniel

Mime
View raw message