lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Uwe Schindler <...@thetaphi.de>
Subject Re: Lucene 6.6: "Too many open files"
Date Tue, 01 Aug 2017 16:35:51 GMT
Hi,

You should reset those settings back to defaults (remove the inner settings in the factory).
30 merge threads will eat up all your file handles. In earlier versions of Lucene, internal
limitations in IndexWriter make it unlikely, that you spawn too many threads, so 30 had no
effect.

In Lucene 6, the number of merges and threads are automatically choosen by your disk type
(SSD detection) and CPU count. So you should definitely use defaults first and only ever change
it for good reasons (if told you by specialists).

Uwe

Am 1. August 2017 17:25:43 MESZ schrieb Nawab Zada Asad Iqbal <khichi@gmail.com>:
>Thanks Jigar
>I haven't tweaked ConcurrentMergeStrategy between 4.5.0 and 6.6. Here
>is
>what I have:
>
><mergeScheduler
>class="org.apache.lucene.index.ConcurrentMergeScheduler">
>  <int name="maxThreadCount">30</int>
>  <int name="maxMergeCount">30</int>
></mergeScheduler>
>
>
>On Mon, Jul 31, 2017 at 8:56 PM, Jigar Shah <jigaronline@gmail.com>
>wrote:
>
>> I faced such problem when I used nomergepolicy, and did some code to
>manual
>> merging segments which had bug and I had same issue. Make sure you
>have
>> default AFAIR ConcurrentMergeStrategy enabled. And its is configured
>with
>> appropriate settings.
>>
>> On Jul 31, 2017 11:21 PM, "Erick Erickson" <erickerickson@gmail.com>
>> wrote:
>>
>> > No, nothing's changed fundamentally. But you say:
>> >
>> > "We have some batch indexing scripts, which
>> > flood the solr servers with indexing requests (while keeping
>> open-searcher
>> > false)"
>> >
>> > What is your commit interval? Regardless of whether openSearcher is
>false
>> > or not, background merging continues apace with every commit. By
>any
>> chance
>> > did you change your merge policy (or not copy the one from 4x to
>6x)?
>> Shot
>> > in the dark...
>> >
>> > Best,
>> > Erick
>> >
>> > On Mon, Jul 31, 2017 at 7:15 PM, Nawab Zada Asad Iqbal
><khichi@gmail.com
>> >
>> > wrote:
>> > > Hi,
>> > >
>> > > I am upgrading from solr4.5 to solr6.6 and hitting this issue
>during
>> > > complete reindexing scenario.  We have some batch indexing
>scripts,
>> which
>> > > flood the solr servers with indexing requests (while keeping
>> > open-searcher
>> > > false) for many hours and then perform one commit. This used to
>work
>> fine
>> > > with 4.5, but with 6.6, i get 'Too many open files' within a
>couple of
>> > > minutes. I have checked that "ulimit" is same between old and new
>> > servers.
>> > >
>> > > Has something fundamentally changed in recent lucene versions,
>which
>> > keeps
>> > > file descriptors around for a longer time?
>> > >
>> > >
>> > > Here is a sample error message:
>> > >     at org.apache.lucene.index.IndexWriter.ensureOpen(
>> > IndexWriter.java:749)
>> > >     at org.apache.lucene.index.IndexWriter.ensureOpen(
>> > IndexWriter.java:763)
>> > >     at org.apache.lucene.index.IndexWriter.commit(
>> IndexWriter.java:3206)
>> > >     at
>> > > org.apache.solr.update.DirectUpdateHandler2.commit(
>> > DirectUpdateHandler2.java:644)
>> > >     at
>> > >
>org.apache.solr.update.processor.RunUpdateProcessor.processCommit(
>> > RunUpdateProcessorFactory.java:93)
>> > >     at
>> > >
>org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(
>> > UpdateRequestProcessor.java:68)
>> > >     at
>> > > org.apache.solr.update.processor.DistributedUpdateProcessor.
>> > doLocalCommit(DistributedUpdateProcessor.java:1894)
>> > >     at
>> > > org.apache.solr.update.processor.DistributedUpdateProcessor.
>> > processCommit(DistributedUpdateProcessor.java:1871)
>> > >     at
>> > > org.apache.solr.update.processor.LogUpdateProcessorFactory$
>> >
>LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
>> > >     at
>> > >
>org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(
>> > UpdateRequestProcessor.java:68)
>> > >     at
>> > > org.apache.solr.handler.RequestHandlerUtils.handleCommit(
>> > RequestHandlerUtils.java:68)
>> > >     at
>> > >
>org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(
>> > ContentStreamHandlerBase.java:62)
>> > >     at
>> > > org.apache.solr.handler.RequestHandlerBase.handleRequest(
>> > RequestHandlerBase.java:173)
>> > >     at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
>> > >     at org.apache.solr.servlet.HttpSolrCall.execute(
>> > HttpSolrCall.java:723)
>> > >     at org.apache.solr.servlet.HttpSolrCall.call(
>> HttpSolrCall.java:529)
>> > >     at
>> > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> > SolrDispatchFilter.java:361)
>> > >     at
>> > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> > SolrDispatchFilter.java:305)
>> > >     at
>> > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> > doFilter(ServletHandler.java:1691)
>> > >     at
>> > > org.eclipse.jetty.servlet.ServletHandler.doHandle(
>> > ServletHandler.java:582)
>> > >     at
>> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> > ScopedHandler.java:143)
>> > >     at
>> > > org.eclipse.jetty.security.SecurityHandler.handle(
>> > SecurityHandler.java:548)
>> > >     at
>> > > org.eclipse.jetty.server.session.SessionHandler.
>> > doHandle(SessionHandler.java:226)
>> > >     at
>> > > org.eclipse.jetty.server.handler.ContextHandler.
>> > doHandle(ContextHandler.java:1180)
>> > >     at
>> > > org.eclipse.jetty.servlet.ServletHandler.doScope(
>> > ServletHandler.java:512)
>> > >     at
>> > > org.eclipse.jetty.server.session.SessionHandler.
>> > doScope(SessionHandler.java:185)
>> > >     at
>> > > org.eclipse.jetty.server.handler.ContextHandler.
>> > doScope(ContextHandler.java:1112)
>> > >     at
>> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> > ScopedHandler.java:141)
>> > >     at
>> > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
>> > ContextHandlerCollection.java:213)
>> > >     at
>> > > org.eclipse.jetty.server.handler.HandlerCollection.
>> > handle(HandlerCollection.java:119)
>> > >     at
>> > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> > HandlerWrapper.java:134)
>> > >     at
>> > > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
>> > RewriteHandler.java:335)
>> > >     at
>> > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> > HandlerWrapper.java:134)
>> > >     at org.eclipse.jetty.server.Server.handle(Server.java:534)
>> > >     at org.eclipse.jetty.server.HttpChannel.handle(
>> HttpChannel.java:320)
>> > >     at
>> > > org.eclipse.jetty.server.HttpConnection.onFillable(
>> > HttpConnection.java:251)
>> > >     at
>> > > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
>> > AbstractConnection.java:273)
>> > >     at org.eclipse.jetty.io.FillInterest.fillable(
>> FillInterest.java:95)
>> > >     at
>> > > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
>> > SelectChannelEndPoint.java:93)
>> > >     at
>> > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
>> > executeProduceConsume(ExecuteProduceConsume.java:303)
>> > >     at
>> > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
>> > produceConsume(ExecuteProduceConsume.java:148)
>> > >     at
>> > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
>> > ExecuteProduceConsume.java:136)
>> > >     at
>> > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> > QueuedThreadPool.java:671)
>> > >     at
>> > > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>> > QueuedThreadPool.java:589)
>> > >     at java.lang.Thread.run(Thread.java:748)
>> > > Caused by: java.nio.file.FileSystemException:
>> > > /local/var/solr/shard2/filesearch/data/index/_34w5.fdx: Too many
>open
>> > files
>> > >     at
>> > >
>sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
>> > >     at sun.nio.fs.UnixException.rethrowAsIOException(
>> > UnixException.java:102)
>> > >     at sun.nio.fs.UnixException.rethrowAsIOException(
>> > UnixException.java:107)
>> > >     at
>> > > sun.nio.fs.UnixFileSystemProvider.newByteChannel(
>> > UnixFileSystemProvider.java:214)
>> > >     at
>> > > java.nio.file.spi.FileSystemProvider.newOutputStream(
>> > FileSystemProvider.java:434)
>> > >     at java.nio.file.Files.newOutputStream(Files.java:216)
>> > >     at
>> > > org.apache.lucene.store.FSDirectory$FSIndexOutput.&lt;
>> > init&gt;(FSDirectory.java:413)
>> > >     at
>> > > org.apache.lucene.store.FSDirectory$FSIndexOutput.&lt;
>> > init&gt;(FSDirectory.java:409)
>> > >     at
>> > >
>org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
>> > >     at
>> > >
>org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(
>> > LockValidatingDirectoryWrapper.java:44)
>> > >     at
>> > > org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(
>> > TrackingDirectoryWrapper.java:43)
>> > >     at
>> > >
>org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.
>> > &lt;init&gt;(CompressingStoredFieldsWriter.java:114)
>> > >     at
>> > >
>org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.
>> > fieldsWriter(CompressingStoredFieldsFormat.java:128)
>> > >     at
>> > > org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.
>> > fieldsWriter(Lucene50StoredFieldsFormat.java:183)
>> > >     at
>> > >
>org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(
>> > StoredFieldsConsumer.java:39)
>> > >     at
>> > > org.apache.lucene.index.StoredFieldsConsumer.startDocument(
>> > StoredFieldsConsumer.java:46)
>> > >     at
>> > > org.apache.lucene.index.DefaultIndexingChain.startStoredFields(
>> > DefaultIndexingChain.java:364)
>> > >     at
>> > > org.apache.lucene.index.DefaultIndexingChain.processDocument(
>> > DefaultIndexingChain.java:398)
>> > >     at
>> > > org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(
>> > DocumentsWriterPerThread.java:232)
>> > >     at
>> > > org.apache.lucene.index.DocumentsWriter.updateDocument(
>> > DocumentsWriter.java:478)
>> > >     at
>> > > org.apache.lucene.index.IndexWriter.updateDocument(
>> > IndexWriter.java:1571)
>> > >     at
>> > > org.apache.solr.update.DirectUpdateHandler2.updateDocument(
>> > DirectUpdateHandler2.java:924)
>> > >     at
>> > > org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(
>> > DirectUpdateHandler2.java:913)
>> > >     at
>> > > org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(
>> > DirectUpdateHandler2.java:302)
>> > >     at
>> > > org.apache.solr.update.DirectUpdateHandler2.addDoc0(
>> > DirectUpdateHandler2.java:239)
>> > >     at
>> > > org.apache.solr.update.DirectUpdateHandler2.addDoc(
>> > DirectUpdateHandler2.java:194)
>> > >     at
>> > > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(
>> > RunUpdateProcessorFactory.java:67)
>> > >     at
>> > >
>org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(
>> > UpdateRequestProcessor.java:55)
>> > >     at
>> > > org.apache.solr.update.processor.DistributedUpdateProcessor.
>> versionAdd(
>> > DistributedUpdateProcessor.java:1005)
>> > >     at
>> > > org.apache.solr.update.processor.DistributedUpdateProcessor.
>> processAdd(
>> > DistributedUpdateProcessor.java:748)
>> > >     at
>> > > org.apache.solr.update.processor.LogUpdateProcessorFactory$
>> > LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>> > >     at
>> > >
>org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(
>> > UpdateRequestProcessor.java:55)
>> > >     at
>> > > org.apache.solr.update.processor.LanguageIdentifierUpdateProces
>> > sor.processAdd(LanguageIdentifierUpdateProcessor.java:205)
>> > >     at
>> > >
>org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:
>> > 261)
>> > >     at org.apache.solr.handler.loader.XMLLoader.load(
>> XMLLoader.java:188)
>> > >     at
>> > > org.apache.solr.handler.UpdateRequestHandler$1.load(
>> > UpdateRequestHandler.java:97)
>> > >     at
>> > >
>org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(
>> > ContentStreamHandlerBase.java:68)
>> > >     ... 33 more
>> > >
>> > >
>> > >
>> > > Thanks
>> > > Nawab
>> >
>> >
>---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>> >
>>

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message