lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: severe problems with soft and hard commits in a large index
Date Wed, 06 May 2015 15:19:25 GMT
On 5/6/2015 8:55 AM, adfel70 wrote:
> Thank you for the detailed answer.
> How can I decrease the impact of opening a searcher in such a large index?
> especially the impact of heap usage that causes OOM.

See the wiki link I sent.  It talks about some of the things that
require a lot of heap and ways you can reduce those requirements.  The
lists are nowhere near complete.

http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

> regarding GC tuning - I am doint that.
> here are the params I use:
> AggresiveOpts
> UseLargePages
> ParallelRefProcEnabled
> CMSParallelRemarkEnabled
> CMSMaxAbortablePrecleanTime=6000
> CMDTriggerPermRatio=80
> CMSInitiatingOccupancyFraction=70
> UseCMSInitiatinOccupancyOnly
> CMSFullGCsBeforeCompaction=1
> PretenureSizeThreshold=64m
> CMSScavengeBeforeRemark
> UseConcMarkSweepGC
> MaxTenuringThreshold=8
> TargetSurvivorRatio=90
> SurviorRatio=4
> NewRatio=2
> Xms16gb
> Xmn28gb

This list seems to have come from re-typing the GC options.  If this is
a cut/paste, I would not expect it to work -- there are typos and part
of each option is missing other characters.  Assuming that this is not
cut/paste, it is mostly similar to the CMS options that I once used for
my own index:

http://wiki.apache.org/solr/ShawnHeisey#CMS_.28ConcurrentMarkSweep.29_Collector

> How many documents per shard are recommended?
> Note that I use nested documents. total collection size is 3 billion docs,
> number of parent docs is 600 million. the rest are children.

For the G1 collector, you'd want to limit each shard to about 100
million docs.  I have no idea about limitations and capabilities where
very large memory allocations are concerned with the CMS collector. 
Running the latest Java 8 is *strongly* recommended, no matter what
collector you're using, because recent versions have incorporated GC
improvements with large memory allocations.  With Java 8u40 and later,
the limitations for 16MB huge allocations on the G1 collector might not
even apply.

Thanks,
Shawn


Mime
View raw message