lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giovanni Fernandez-Kincade <>
Subject RE: Lucene Merge Threads
Date Tue, 13 Oct 2009 02:38:06 GMT
This didn't end up working. I got the following error when I tried to commit:

Oct 12, 2009 8:36:42 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Error loading class '
	at org.apache.solr.core.SolrResourceLoader.findClass(
	at org.apache.solr.core.SolrResourceLoader.newInstance(
	at org.apache.solr.update.SolrIndexWriter.init(
	at org.apache.solr.update.SolrIndexWriter.<init>(
	at org.apache.solr.update.UpdateHandler.createMainIndexWriter(
	at org.apache.solr.update.DirectUpdateHandler2.openWriter(
	at org.apache.solr.update.DirectUpdateHandler2.commit(
	at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(
	at org.apache.solr.handler.XMLLoader.processUpdate(
	at org.apache.solr.handler.XMLLoader.load(
	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(
	at org.apache.solr.core.SolrCore.execute(
	at org.apache.solr.servlet.SolrDispatchFilter.execute(
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(
	at org.apache.catalina.core.StandardWrapperValve.invoke(
	at org.apache.catalina.core.StandardContextValve.invoke(
	at org.apache.catalina.core.StandardHostValve.invoke(
	at org.apache.catalina.valves.ErrorReportValve.invoke(
	at org.apache.catalina.core.StandardEngineValve.invoke(
	at org.apache.catalina.connector.CoyoteAdapter.service(
	at org.apache.coyote.http11.Http11Processor.process(
	at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(
	at org.apache.tomcat.util.threads.ThreadPool$
	at Source)
Caused by: java.lang.ClassNotFoundException: 
	at$ Source)
	at$$YJP$$doPrivileged(Native Method)
	at Source)
	at Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClassInternal(Unknown Source)
	at java.lang.Class.$$YJP$$forName0(Native Method)
	at java.lang.Class.forName0(Unknown Source)
	at java.lang.Class.forName(Unknown Source)
	at org.apache.solr.core.SolrResourceLoader.findClass(
	... 28 more

I believe it's because the MaxThreadCount is not a public property of the ConcurrentMergeSchedulerClass.
You have to call this method to set it:

public void setMaxThreadCount(int count) {
    if (count < 1)
      throw new IllegalArgumentException("count should be at least 1");
    maxThreadCount = count;

Is that possible through the solrconfig?


-----Original Message-----
From: Giovanni Fernandez-Kincade [] 
Sent: Monday, October 12, 2009 7:53 PM
Subject: RE: Lucene Merge Threads

Do you have to make a new call to optimize to make it start the merge again?

-----Original Message-----
From: Jason Rutherglen [] 
Sent: Monday, October 12, 2009 7:28 PM
Subject: Re: Lucene Merge Threads

Try this in solrconfig.xml:

<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
  <int name="maxThreadCount">1</int>

Yes you can stop the process mid-merge.  The partially merged files
will be deleted on restart.

We need to update the wiki?

On Mon, Oct 12, 2009 at 4:05 PM, Giovanni Fernandez-Kincade
<> wrote:
> Hi,
> I'm attempting to optimize a pretty large index, and even though the optimize request
timed out, I watched it using a profiler and saw that the optimize thread continued executing.
Eventually it completed, but in the background I still see a thread performing a merge:
> Lucene Merge Thread #0 [RUNNABLE, IN_NATIVE] CPU time: 17:51
>[], int, int)
>[], int, int)
>$SimpleFSIndexInput.readInternal(byte[], int,
>, FieldInfos)
> org.apache.lucene.index.SegmentMerger.mergeTermInfos(FormatPostingsFieldsConsumer)
> org.apache.lucene.index.SegmentMerger.mergeTerms()
> org.apache.lucene.index.SegmentMerger.merge(boolean)
> org.apache.lucene.index.IndexWriter.mergeMiddle(MergePolicy$OneMerge)
> org.apache.lucene.index.IndexWriter.merge(MergePolicy$OneMerge)
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(MergePolicy$OneMerge)
> org.apache.lucene.index.ConcurrentMergeScheduler$
> This has taken quite a while, and hasn't really been fully utilizing the machine's resources.
After looking at the Lucene source, I noticed that you can set a MaxThreadCount parameter
in this class. Is this parameter exposed by Solr somehow? I see the class mentioned, commented
out, in my solrconfig.xml, but I'm not sure of the correct way to specify the parameter:
> <!--
>     Expert:
>     The Merge Scheduler in Lucene controls how merges are performed.  The ConcurrentMergeScheduler
(Lucene 2.3 default)
>      can perform merges in the background using separate threads.  The SerialMergeScheduler
(Lucene 2.2 default) does not.
>     -->
>    <!--<mergeScheduler>org.apache.lucene.index.ConcurrentMergeScheduler</mergeScheduler>-->
> Also, if I can specify this parameter, is it safe to just start/stop my servlet server
(Tomcat) mid-merge?
> Thanks in advance,
> Gio.

View raw message