lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jaeger, Jay - DOT" <Jay.Jae...@dot.wi.gov>
Subject RE: Performance troubles with solr
Date Wed, 14 Sep 2011 15:07:54 GMT
I don't have enough experience with filter queries to advise well on when to use fq vs. putting
it in the query itself, but I do know that we are not using filter queries, and with index
sizes ranging from 7 Million to 27+ Million we have not seen this kind of issue.

Maybe keeping 16,384 filter queries around, particularly caching the ones with "random age
ranges" is eating your memory up -- so perhaps try moving just that particular fq into q instead
(since it is "random") and just cache the ones where the number of "options" is limited?

What happens if you try your test without the filter queries?  What happens if you put the
additional criteria that are in your filter query into the query itself?

JRJ

-----Original Message-----
From: Yusuf Karakaya [mailto:karakayaysf@gmail.com] 
Sent: Wednesday, September 14, 2011 9:54 AM
To: solr-user@lucene.apache.org
Subject: Re: Performance troubles with solr

Thank you for your reply.
I tried to give most of the information i can but obviously i missed some.
1.  Just what does your "test script" do?   Is it doing updates, or just
queries of the sort you mentioned below?
Test script only sends random queries.
2.  If the test script is doing updates, how are those updates being fed to
Solr?
There are no updates right now, as i failed on performance.
3.  What version of Solr are you running?
I'm using Solr 3.3.0
4.  Why did you increase the default for jetty (around 384m) to 6000m,
particularly given your relatively modest number of documents (2,000,000).
I was trying everything before asking here.
5.  Machine characteristics, particularly operating system and physical
memory on the machine.
OS => Debian 6.0,  Physcal Memory => 32 gb, CPU => 2x Intel Quad Core

On Wed, Sep 14, 2011 at 5:38 PM, Jaeger, Jay - DOT <Jay.Jaeger@dot.wi.gov>wrote:

> I think folks are going to need a *lot* more information.  Particularly
>
> 1.  Just what does your "test script" do?   Is it doing updates, or just
> queries of the sort you mentioned below?
> 2.  If the test script is doing updates, how are those updates being fed to
> Solr?
> 3.  What version of Solr are you running?
> 4.  Why did you increase the default for jetty (around 384m) to 6000m,
> particularly given your relatively modest number of documents (2,000,000).
> 5.  Machine characteristics, particularly operating system and physical
> memory on the machine.
>
> Please refer to http://wiki.apache.org/solr/UsingMailingLists for
> additional guidance in using the mailing list to get help.
>
> -----Original Message-----
> From: Yusuf Karakaya [mailto:karakayaysf@gmail.com]
> Sent: Wednesday, September 14, 2011 9:19 AM
> To: solr-user@lucene.apache.org
> Subject: Performance troubles with solr
>
> Hi, i'm having performance troubles with solr. I don't know if i'm
> expection
> too much from solr or i missconfigured solr.
> When i run a single query its QTime is 500-1000~ ms (without any use of
> caches).
> When i run my test script (with use of caches) QTime increases
> exponentially, reaching 8000~ to 60000~  ms. And Cpu usage also increases
> to
> %550~
>
> My solr-start script:
> java -Duser.timezone=EET -Xmx6000m -jar ./start.jar
>
> 2,000,000~ documents ,  currently there aren't any commits but in future
> there will be 5,000~ updates/additions to documents every 3-5~   min via
> delta import.
>
> Search Query
> sort=userscore+desc
> &start=0
> &q=photo_id:* AND gender:true AND country:MALAWI AND online:false
> &fq=birth:[NOW-31YEARS/DAY TO NOW-17YEARS/DAY]  ( Random age ranges )
> &fq=lastlogin:[* TO NOW-6MONTHS/DAY] ( Only 2 options,   [* TO
> NOW-6MONTHS/DAY] or [NOW-6MONTHS/DAY TO *] )
> &fq=userscore:[500 TO *]  ( Only 2 options, [500 TO *] or [* TO 500] )
> &rows=150
>
> Schema
>
> <field name="id" type="long" indexed="true" stored="true" required="true"/>
> <field name="username" type="string" indexed="true" stored="false"
> required="true"/>
> <field name="namesurname" type="string" indexed="true" stored="false"/>
> <field name="network" type="int" indexed="true" stored="false"/>
> <field name="photo_id" type="int" indexed="true" stored="false"/>
> <field name="gender" type="boolean" indexed="true" stored="false"/>
> <field name="country" type="string" indexed="true" stored="false"/>
> <field name="birth" type="tdate" indexed="true" stored="false"/>
> <field name="lastlogin" type="tdate" indexed="true" stored="false"/>
> <field name="online" type="boolean" indexed="true" stored="false"/>
> <field name="userscore" type="int" indexed="true" stored="false"/>
>
> Cache Sizes & Lazy Load
>
> <filterCache class="solr.FastLRUCache" size="16384" initialSize="4096"
> autowarmCount="4096"/>
> <queryResultCache class="solr.LRUCache" size="16384" initialSize="4096"
> autowarmCount="4096"/>
> <documentCache class="solr.LRUCache" size="16384" initialSize="4096"
> autowarmCount="4096"/>
> <enableLazyFieldLoading>true</enableLazyFieldLoading>
>

Mime
View raw message