lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Date Tue, 19 Oct 2010 20:32:54 GMT

: We will take this approach in our production environment but meanwhile I am
: curious if this issue will be addressed: it seems the new/first searchers do
: not really buy any performance benefits because it uses so much memory,
: especially at core loading time.

There's nothing inheriently wrong with using newSearcher/firstSearcher -- 
for many people they do in fact provide a perf improvement for "real" 
users (at the cost of some initial time spent warming before those users 
ever get access to the searcher)

As i udnerstand it from this thread, your issue is not actually the 
firstSearcher/newSearcher -- your issue (per yonik's ocmments) is that 
with per Segment sorting in 1.4, the FieldCache for some of your fields 
requires a lot more ram in 1.4 then it would have been for Solr 1.3 -- 
which caused GC thrashing during initialization.

Even w/o using firstSearcher/newSearcher, all that RAM is still going to 
be used if/when you do sort on those fields -- all removing the 
firstSearcher/newSearcher queries on those fields has done for you is 
delay when the time spent initializing those FieldCaches happens and when 
that RAM first starts getting used.

It's posisbly you never actual sort on those fields, in which case 
removing those warming queries completely is definitely the way to go -- 
but if you do sort on them then the warming queries can still legitimately 
be helpful (in thta they pay the cost up front before a real user issues 
queries)

As yonik mentioned the real "fix" for the amount of memory being used is 
to switch to the TrieDateFields which use much more efficient FieldCache's 
for sorting -- with that change you can probably start using the warming 
queries again.  (Depending on how you tested, you may not have noticed 
much advantage to having them because you'll really only see the 
advantages on the initial queries that do sorting -- those should show 
huge outlier times w/o the warming queries, but once those poor unlucky 
users have paid the price for initializing hte FieldCache, every one elses 
sorts should be fast)

-Hoss

Mime
View raw message