lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: solr cloud does not start with many collections
Date Wed, 04 Mar 2015 08:02:27 GMT
On 3/3/2015 9:22 PM, Damien Kamerman wrote:
> I've done a similar thing to create the collections. You're going to need
> more memory I think.
> 
> OK, so maxThreads limit on jetty could be causing a distributed dead-lock?

I don't know what the exact problems would be if maxThreads is reached.
 It's probably unpredictable.

With 2674 collections added, 5GB wasn't enough heap.  I started getting
a ton of exceptions during collection creation.  I had to shut down both
Solr instances.  When I brought up the first instance with a 7GB heap
(the one with the embedded zk), it took exactly half an hour for jetty
to start listening on port 8983, and about two hours total for it to
stabilize to the point where everything for that node was green on the
cloud graph.

Even now, nearly three hours after startup, the Solr log is still
spitting out thousands of lines that look like this, so I don't think I
can call it stable:

INFO  - 2015-03-04 07:35:51.166;
org.apache.solr.common.cloud.ZkStateReader; Updating data for mycoll1515
to ver 60

I'm going to try bringing up the other Solr instance now, and if that
stabilizes with all shards in the green, I will try to continue adding
collections.

Side note: I have been able to confirm with these tests that version 5.0
no longer requires increasing jute.maxbuffer to run many collections.
I'm still running with the default value and zookeeper has had no
problems handling all the data.

Thanks,
Shawn


Mime
View raw message