lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nawab Zada Asad Iqbal <khi...@gmail.com>
Subject Re: Lucene 6.6: "Too many open files"
Date Tue, 01 Aug 2017 15:25:43 GMT
Thanks Jigar
I haven't tweaked ConcurrentMergeStrategy between 4.5.0 and 6.6. Here is
what I have:

<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
  <int name="maxThreadCount">30</int>
  <int name="maxMergeCount">30</int>
</mergeScheduler>


On Mon, Jul 31, 2017 at 8:56 PM, Jigar Shah <jigaronline@gmail.com> wrote:

> I faced such problem when I used nomergepolicy, and did some code to manual
> merging segments which had bug and I had same issue. Make sure you have
> default AFAIR ConcurrentMergeStrategy enabled. And its is configured with
> appropriate settings.
>
> On Jul 31, 2017 11:21 PM, "Erick Erickson" <erickerickson@gmail.com>
> wrote:
>
> > No, nothing's changed fundamentally. But you say:
> >
> > "We have some batch indexing scripts, which
> > flood the solr servers with indexing requests (while keeping
> open-searcher
> > false)"
> >
> > What is your commit interval? Regardless of whether openSearcher is false
> > or not, background merging continues apace with every commit. By any
> chance
> > did you change your merge policy (or not copy the one from 4x to 6x)?
> Shot
> > in the dark...
> >
> > Best,
> > Erick
> >
> > On Mon, Jul 31, 2017 at 7:15 PM, Nawab Zada Asad Iqbal <khichi@gmail.com
> >
> > wrote:
> > > Hi,
> > >
> > > I am upgrading from solr4.5 to solr6.6 and hitting this issue during
> > > complete reindexing scenario.  We have some batch indexing scripts,
> which
> > > flood the solr servers with indexing requests (while keeping
> > open-searcher
> > > false) for many hours and then perform one commit. This used to work
> fine
> > > with 4.5, but with 6.6, i get 'Too many open files' within a couple of
> > > minutes. I have checked that "ulimit" is same between old and new
> > servers.
> > >
> > > Has something fundamentally changed in recent lucene versions, which
> > keeps
> > > file descriptors around for a longer time?
> > >
> > >
> > > Here is a sample error message:
> > >     at org.apache.lucene.index.IndexWriter.ensureOpen(
> > IndexWriter.java:749)
> > >     at org.apache.lucene.index.IndexWriter.ensureOpen(
> > IndexWriter.java:763)
> > >     at org.apache.lucene.index.IndexWriter.commit(
> IndexWriter.java:3206)
> > >     at
> > > org.apache.solr.update.DirectUpdateHandler2.commit(
> > DirectUpdateHandler2.java:644)
> > >     at
> > > org.apache.solr.update.processor.RunUpdateProcessor.processCommit(
> > RunUpdateProcessorFactory.java:93)
> > >     at
> > > org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(
> > UpdateRequestProcessor.java:68)
> > >     at
> > > org.apache.solr.update.processor.DistributedUpdateProcessor.
> > doLocalCommit(DistributedUpdateProcessor.java:1894)
> > >     at
> > > org.apache.solr.update.processor.DistributedUpdateProcessor.
> > processCommit(DistributedUpdateProcessor.java:1871)
> > >     at
> > > org.apache.solr.update.processor.LogUpdateProcessorFactory$
> > LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
> > >     at
> > > org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(
> > UpdateRequestProcessor.java:68)
> > >     at
> > > org.apache.solr.handler.RequestHandlerUtils.handleCommit(
> > RequestHandlerUtils.java:68)
> > >     at
> > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(
> > ContentStreamHandlerBase.java:62)
> > >     at
> > > org.apache.solr.handler.RequestHandlerBase.handleRequest(
> > RequestHandlerBase.java:173)
> > >     at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> > >     at org.apache.solr.servlet.HttpSolrCall.execute(
> > HttpSolrCall.java:723)
> > >     at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:529)
> > >     at
> > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> > SolrDispatchFilter.java:361)
> > >     at
> > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> > SolrDispatchFilter.java:305)
> > >     at
> > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> > doFilter(ServletHandler.java:1691)
> > >     at
> > > org.eclipse.jetty.servlet.ServletHandler.doHandle(
> > ServletHandler.java:582)
> > >     at
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(
> > ScopedHandler.java:143)
> > >     at
> > > org.eclipse.jetty.security.SecurityHandler.handle(
> > SecurityHandler.java:548)
> > >     at
> > > org.eclipse.jetty.server.session.SessionHandler.
> > doHandle(SessionHandler.java:226)
> > >     at
> > > org.eclipse.jetty.server.handler.ContextHandler.
> > doHandle(ContextHandler.java:1180)
> > >     at
> > > org.eclipse.jetty.servlet.ServletHandler.doScope(
> > ServletHandler.java:512)
> > >     at
> > > org.eclipse.jetty.server.session.SessionHandler.
> > doScope(SessionHandler.java:185)
> > >     at
> > > org.eclipse.jetty.server.handler.ContextHandler.
> > doScope(ContextHandler.java:1112)
> > >     at
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(
> > ScopedHandler.java:141)
> > >     at
> > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> > ContextHandlerCollection.java:213)
> > >     at
> > > org.eclipse.jetty.server.handler.HandlerCollection.
> > handle(HandlerCollection.java:119)
> > >     at
> > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> > HandlerWrapper.java:134)
> > >     at
> > > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
> > RewriteHandler.java:335)
> > >     at
> > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> > HandlerWrapper.java:134)
> > >     at org.eclipse.jetty.server.Server.handle(Server.java:534)
> > >     at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:320)
> > >     at
> > > org.eclipse.jetty.server.HttpConnection.onFillable(
> > HttpConnection.java:251)
> > >     at
> > > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> > AbstractConnection.java:273)
> > >     at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
> > >     at
> > > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> > SelectChannelEndPoint.java:93)
> > >     at
> > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> > executeProduceConsume(ExecuteProduceConsume.java:303)
> > >     at
> > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> > produceConsume(ExecuteProduceConsume.java:148)
> > >     at
> > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> > ExecuteProduceConsume.java:136)
> > >     at
> > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> > QueuedThreadPool.java:671)
> > >     at
> > > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> > QueuedThreadPool.java:589)
> > >     at java.lang.Thread.run(Thread.java:748)
> > > Caused by: java.nio.file.FileSystemException:
> > > /local/var/solr/shard2/filesearch/data/index/_34w5.fdx: Too many open
> > files
> > >     at
> > > sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> > >     at sun.nio.fs.UnixException.rethrowAsIOException(
> > UnixException.java:102)
> > >     at sun.nio.fs.UnixException.rethrowAsIOException(
> > UnixException.java:107)
> > >     at
> > > sun.nio.fs.UnixFileSystemProvider.newByteChannel(
> > UnixFileSystemProvider.java:214)
> > >     at
> > > java.nio.file.spi.FileSystemProvider.newOutputStream(
> > FileSystemProvider.java:434)
> > >     at java.nio.file.Files.newOutputStream(Files.java:216)
> > >     at
> > > org.apache.lucene.store.FSDirectory$FSIndexOutput.&lt;
> > init&gt;(FSDirectory.java:413)
> > >     at
> > > org.apache.lucene.store.FSDirectory$FSIndexOutput.&lt;
> > init&gt;(FSDirectory.java:409)
> > >     at
> > > org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
> > >     at
> > > org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(
> > LockValidatingDirectoryWrapper.java:44)
> > >     at
> > > org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(
> > TrackingDirectoryWrapper.java:43)
> > >     at
> > > org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.
> > &lt;init&gt;(CompressingStoredFieldsWriter.java:114)
> > >     at
> > > org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.
> > fieldsWriter(CompressingStoredFieldsFormat.java:128)
> > >     at
> > > org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.
> > fieldsWriter(Lucene50StoredFieldsFormat.java:183)
> > >     at
> > > org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(
> > StoredFieldsConsumer.java:39)
> > >     at
> > > org.apache.lucene.index.StoredFieldsConsumer.startDocument(
> > StoredFieldsConsumer.java:46)
> > >     at
> > > org.apache.lucene.index.DefaultIndexingChain.startStoredFields(
> > DefaultIndexingChain.java:364)
> > >     at
> > > org.apache.lucene.index.DefaultIndexingChain.processDocument(
> > DefaultIndexingChain.java:398)
> > >     at
> > > org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(
> > DocumentsWriterPerThread.java:232)
> > >     at
> > > org.apache.lucene.index.DocumentsWriter.updateDocument(
> > DocumentsWriter.java:478)
> > >     at
> > > org.apache.lucene.index.IndexWriter.updateDocument(
> > IndexWriter.java:1571)
> > >     at
> > > org.apache.solr.update.DirectUpdateHandler2.updateDocument(
> > DirectUpdateHandler2.java:924)
> > >     at
> > > org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(
> > DirectUpdateHandler2.java:913)
> > >     at
> > > org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(
> > DirectUpdateHandler2.java:302)
> > >     at
> > > org.apache.solr.update.DirectUpdateHandler2.addDoc0(
> > DirectUpdateHandler2.java:239)
> > >     at
> > > org.apache.solr.update.DirectUpdateHandler2.addDoc(
> > DirectUpdateHandler2.java:194)
> > >     at
> > > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(
> > RunUpdateProcessorFactory.java:67)
> > >     at
> > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(
> > UpdateRequestProcessor.java:55)
> > >     at
> > > org.apache.solr.update.processor.DistributedUpdateProcessor.
> versionAdd(
> > DistributedUpdateProcessor.java:1005)
> > >     at
> > > org.apache.solr.update.processor.DistributedUpdateProcessor.
> processAdd(
> > DistributedUpdateProcessor.java:748)
> > >     at
> > > org.apache.solr.update.processor.LogUpdateProcessorFactory$
> > LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> > >     at
> > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(
> > UpdateRequestProcessor.java:55)
> > >     at
> > > org.apache.solr.update.processor.LanguageIdentifierUpdateProces
> > sor.processAdd(LanguageIdentifierUpdateProcessor.java:205)
> > >     at
> > > org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:
> > 261)
> > >     at org.apache.solr.handler.loader.XMLLoader.load(
> XMLLoader.java:188)
> > >     at
> > > org.apache.solr.handler.UpdateRequestHandler$1.load(
> > UpdateRequestHandler.java:97)
> > >     at
> > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(
> > ContentStreamHandlerBase.java:68)
> > >     ... 33 more
> > >
> > >
> > >
> > > Thanks
> > > Nawab
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message