lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Query takes a long time Solr 6.1.0
Date Fri, 07 Jun 2019 15:30:12 GMT
On 6/6/2019 5:45 AM, vishal patel wrote:
> One server(256GB RAM) has two below Solr instance and other application also
> 1) shards1 (80GB heap ,790GB Storage, 449GB Indexed data)
> 2) replica of shard2 (80GB heap, 895GB Storage, 337GB Indexed data)
> 
> The second server(256GB RAM and 1 TB storage) has two below Solr instance and other application
also
> 1) shards2 (80GB heap, 790GB Storage, 338GB Indexed data)
> 2) replica of shard1 (80GB heap, 895GB Storage, 448GB Indexed data)

An 80GB heap is ENORMOUS.  And you have two of those per server.  Do you 
*know* that you need a heap that large?  You only have 50 million 
documents total, two instances that each have 80GB seems completely 
unnecessary.  I would think that one instance with a much smaller heap 
would handle just about anything you could throw at 50 million documents.

With 160GB taken by heaps, you're leaving less than 100GB of memory to 
cache over 700GB of index.  This is not going to work well, especially 
if your index doesn't have many fields that are stored.  It will cause a 
lot of disk I/O.

> Both server memory and disk usage:
> https://drive.google.com/drive/folders/11GoZy8C0i-qUGH-ranPD8PCoPWCxeS-5

Unless you have changed the DirectoryFactory to something that's not 
default, your process listing does not reflect over 700GB of index data. 
  If you have changed the DirectoryFactory, then I would strongly 
recommend removing that part of your config and letting Solr use its 
default.

> Note: Average 40GB heap used normally in each Solr instance. when replica gets down at
that time disk IO are high and also GC pause time above 15 seconds. We can not identify the
exact issue of replica recovery OR down from logs. due to the GC pause? OR due to disk IO
high? OR due to time-consuming query? OR due to heavy indexing?

With an 80GB heap, I'm not really surprised you're seeing GC pauses 
above 15 seconds.  I have seen pauses that long with a heap that's only 8GB.

GC pauses lasting that long will cause problems with SolrCloud.  Nodes 
going into recovery is common.

Thanks,
Shawn

Mime
View raw message