lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sudip Mukherjee <smukher...@commvault.com>
Subject RE: Replicas do not come up after nodes are restarted in SOLR cloud
Date Fri, 07 Sep 2018 10:16:15 GMT
Hi Shawn,

Thanks for the insights. 

I verified the points you mentioned,
Socket timeout defaults weren't changed.

Both nodes (two different hosts, windows OS ) are given heap space of 4GB. They are having
two Collections as of now.
One is without replicas but with 8 shards on each node.
One is with replicas and 1 shard. (both replicas are down)

I monitored the application for some time and I do not see obvious memory issues or prolonged
garbage collection.

Also, the nodes are registered with their hostnames in the collection.


Thanks,
Sudip

-----Original Message-----
From: Shawn Heisey [mailto:elyograg@elyograg.org] 
Sent: Wednesday, September 05, 2018 7:23 PM
To: solr-user@lucene.apache.org
Subject: Re: Replicas do not come up after nodes are restarted in SOLR cloud

On 9/5/2018 2:55 AM, Sudip Mukherjee wrote:
> I have a 2 node SOLR (7.x) cloud cluster on which I have collection 
> with replicas ( replicationFactor = 2, shard = 1 ). I am seeing that the replicas do
not come up ( state is "down")  when both nodes are restarted. From the "legend" in Graph
section, I see that the replicas are in "recovery failed" state.
<snip>
> Caused by: java.net.SocketTimeoutException: Read timed out
>                  at java.net.SocketInputStream.socketRead0(Native Method)
>                  at 
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

Have you changed the socket timeout in Solr's config?

The socket timeout for internode requests defaults to 60 seconds.  If something happened
that prevented a Solr server from responding within
60 seconds, then there's something *REALLY* wrong.

My best guess is that your Solr heap is too small, causing Java to spend almost all of its
time doing garbage collection.  Or that a too-small heap has caused one of your servers to
experience an OutOfMemoryError, which on non-Windows systems will result in the Solr process
being killed.

Some questions in case that's not it:

How many collections do you have on this setup?

In the admin UI (Cloud tab), what hostname do your nodes show they are registered as?  If
it's localhost, that's going to be a problem for a 2-node cluster.

Thanks,
Shawn

***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
Mime
View raw message