lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giovanni Fernandez-Kincade <gfernandez-kinc...@capitaliq.com>
Subject RE: Lucene Merge Threads
Date Tue, 13 Oct 2009 14:49:54 GMT
Here's the version information from the admin page:

Solr Specification Version: 1.3.0.2009.07.28.18.51.06
Solr Implementation Version: 1.4-dev ${svnversion} - gkincade - 2009-07-28 18:51:06
Lucene Specification Version: 2.9-dev
Lucene Implementation Version: 2.9-dev 794238 - 2009-07-15 18:05:08




-----Original Message-----
From: noble.paul@gmail.com [mailto:noble.paul@gmail.com] On Behalf Of Noble Paul ??????? ??????
Sent: Tuesday, October 13, 2009 2:32 AM
To: solr-user@lucene.apache.org
Subject: Re: Lucene Merge Threads

which version of Solr are you using? the <int
name="maxThreadCount">1</int> syntax was added recently

On Tue, Oct 13, 2009 at 8:08 AM, Giovanni Fernandez-Kincade
<gfernandez-kincade@capitaliq.com> wrote:
> This didn't end up working. I got the following error when I tried to commit:
>
> Oct 12, 2009 8:36:42 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Error loading class '
>                5
>        '
>        at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:310)
>        at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:325)
>        at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:81)
>        at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:178)
>        at org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123)
>        at org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:172)
>        at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:400)
>        at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>        at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
>        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
>        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
>        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
>        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
>        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
>        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
>        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
>        at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.ClassNotFoundException:
>                5
>
>        at java.net.URLClassLoader$1.run(Unknown Source)
>        at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
>        at java.security.AccessController.doPrivileged(Unknown Source)
>        at java.net.URLClassLoader.findClass(Unknown Source)
>        at java.lang.ClassLoader.loadClass(Unknown Source)
>        at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
>        at java.lang.ClassLoader.loadClass(Unknown Source)
>        at java.lang.ClassLoader.loadClassInternal(Unknown Source)
>        at java.lang.Class.$$YJP$$forName0(Native Method)
>        at java.lang.Class.forName0(Unknown Source)
>        at java.lang.Class.forName(Unknown Source)
>        at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:294)
>        ... 28 more
>
>
> I believe it's because the MaxThreadCount is not a public property of the ConcurrentMergeSchedulerClass.
You have to call this method to set it:
>
> public void setMaxThreadCount(int count) {
>    if (count < 1)
>      throw new IllegalArgumentException("count should be at least 1");
>    maxThreadCount = count;
>  }
>
> Is that possible through the solrconfig?
>
> Thanks,
> Gio.
>
> -----Original Message-----
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kincade@capitaliq.com]
> Sent: Monday, October 12, 2009 7:53 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Lucene Merge Threads
>
> Do you have to make a new call to optimize to make it start the merge again?
>
> -----Original Message-----
> From: Jason Rutherglen [mailto:jason.rutherglen@gmail.com]
> Sent: Monday, October 12, 2009 7:28 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Lucene Merge Threads
>
> Try this in solrconfig.xml:
>
> <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
>  <int name="maxThreadCount">1</int>
> </mergeScheduler>
>
> Yes you can stop the process mid-merge.  The partially merged files
> will be deleted on restart.
>
> We need to update the wiki?
>
> On Mon, Oct 12, 2009 at 4:05 PM, Giovanni Fernandez-Kincade
> <gfernandez-kincade@capitaliq.com> wrote:
>> Hi,
>> I'm attempting to optimize a pretty large index, and even though the optimize request
timed out, I watched it using a profiler and saw that the optimize thread continued executing.
Eventually it completed, but in the background I still see a thread performing a merge:
>>
>> Lucene Merge Thread #0 [RUNNABLE, IN_NATIVE] CPU time: 17:51
>> java.io.RandomAccessFile.readBytes(byte[], int, int)
>> java.io.RandomAccessFile.read(byte[], int, int)
>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
int, int)
>> org.apache.lucene.store.BufferedIndexInput.refill()
>> org.apache.lucene.store.BufferedIndexInput.readByte()
>> org.apache.lucene.store.IndexInput.readVInt()
>> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
>> org.apache.lucene.index.SegmentTermEnum.next()
>> org.apache.lucene.index.SegmentMergeInfo.next()
>> org.apache.lucene.index.SegmentMerger.mergeTermInfos(FormatPostingsFieldsConsumer)
>> org.apache.lucene.index.SegmentMerger.mergeTerms()
>> org.apache.lucene.index.SegmentMerger.merge(boolean)
>> org.apache.lucene.index.IndexWriter.mergeMiddle(MergePolicy$OneMerge)
>> org.apache.lucene.index.IndexWriter.merge(MergePolicy$OneMerge)
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(MergePolicy$OneMerge)
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run()
>>
>>
>> This has taken quite a while, and hasn't really been fully utilizing the machine's
resources. After looking at the Lucene source, I noticed that you can set a MaxThreadCount
parameter in this class. Is this parameter exposed by Solr somehow? I see the class mentioned,
commented out, in my solrconfig.xml, but I'm not sure of the correct way to specify the parameter:
>>
>> <!--
>>     Expert:
>>     The Merge Scheduler in Lucene controls how merges are performed.  The ConcurrentMergeScheduler
(Lucene 2.3 default)
>>      can perform merges in the background using separate threads.  The SerialMergeScheduler
(Lucene 2.2 default) does not.
>>     -->
>>    <!--<mergeScheduler>org.apache.lucene.index.ConcurrentMergeScheduler</mergeScheduler>-->
>>
>>
>> Also, if I can specify this parameter, is it safe to just start/stop my servlet server
(Tomcat) mid-merge?
>>
>> Thanks in advance,
>> Gio.
>>
>



-- 
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Mime
View raw message