lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Trym R. Møller" <t...@sigmat.dk>
Subject Re: Solr Hanging
Date Mon, 23 Apr 2012 09:37:48 GMT
Hi

I have succeeded in reproducing the scenario with two Solr instances 
running. They cover a single collection with two slices and two replica, 
two cores in each Solr instance. I have changed the number of threads 
that Jetty is allowed to use as follows:
<New class="org.mortbay.thread.QueuedThreadPool">
<Set name="minThreads">3</Set>
<Set name="maxThreads">3</Set>
<Set name="lowThreads">0</Set>
</New>
And when indexing a single document this works fine but when 
concurrently indexing 10 documents, Solr frequently hangs.
I know that Jetty per default are allowed to use 10.000 threads, but in 
my other setup, all these 10.000 allowed thread are used on a single 
Solr instance (I have 7 Solr instances) after some days and the hanging 
scenario occurs.

I'm not sure if just adjusting the allowed number of threads are the 
best solution and would like to get some input as what to expect and if 
there are other things I can adjust.
My setup is as written before 7 Solr instances handling a single 
collection with 28 leaders and 28 replicas distributed fairly on the 
Solrs (8 cores on each Solr).

Thanks for any input.

Best regards Trym


Den 19-04-2012 14:36, Yonik Seeley skrev:
> On Thu, Apr 19, 2012 at 4:25 AM, "Trym R. Møller"<trym@sigmat.dk>  wrote:
>> Hi
>>
>> I am using Solr trunk and have 7 Solr instances running with 28 leaders and
>> 28 replicas for a single collection.
>> After indexing a while (a couple of days) the solrs start hanging and doing
>> a thread dump on the jvm I see blocked threads like the following:
>>     Thread 2369: (state = BLOCKED)
>>      - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
>> information may be imprecise)
>>      - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
>> line=158 (Compiled frame)
>>      -
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
>> @bci=42, line=1987 (Compiled frame)
>>      - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
>> (Compiled frame)
>>      - java.util.concurrent.ExecutorCompletionService.take() @bci=4, line=164
>> (Compiled frame)
>>      - org.apache.solr.update.SolrCmdDistributor.checkResponses(boolean)
>> @bci=27, line=350 (Compiled frame)
>>      - org.apache.solr.update.SolrCmdDistributor.finish() @bci=18, line=98
>> (Compiled frame)
>>      - org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish()
>> @bci=4, line=299 (Compiled frame)
>>      - org.apache.solr.update.processor.DistributedUpdateProcessor.finish()
>> @bci=1, line=817 (Compiled frame)
>>     ...
>>      - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=25, line=582
>> (Interpreted frame)
>>
>> I read the stack trace as my indexing client has indexed a document and this
>> Solr is now waiting for the replica? to respond before returning an answer
>> to the client.
> Correct.  What's the full stack trace like on both a leader and replica?
> We need to know what the replica is blocking on.
>
> What version of trunk are you using?
>
> -Yonik
> lucenerevolution.com - Lucene/Solr Open Source Search Conference.
> Boston May 7-10

Mime
View raw message