lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Lucene 6.6: "Too many open files"
Date Tue, 01 Aug 2017 03:20:47 GMT
No, nothing's changed fundamentally. But you say:

"We have some batch indexing scripts, which
flood the solr servers with indexing requests (while keeping open-searcher
false)"

What is your commit interval? Regardless of whether openSearcher is false
or not, background merging continues apace with every commit. By any chance
did you change your merge policy (or not copy the one from 4x to 6x)? Shot
in the dark...

Best,
Erick

On Mon, Jul 31, 2017 at 7:15 PM, Nawab Zada Asad Iqbal <khichi@gmail.com> wrote:
> Hi,
>
> I am upgrading from solr4.5 to solr6.6 and hitting this issue during
> complete reindexing scenario.  We have some batch indexing scripts, which
> flood the solr servers with indexing requests (while keeping open-searcher
> false) for many hours and then perform one commit. This used to work fine
> with 4.5, but with 6.6, i get 'Too many open files' within a couple of
> minutes. I have checked that "ulimit" is same between old and new servers.
>
> Has something fundamentally changed in recent lucene versions, which keeps
> file descriptors around for a longer time?
>
>
> Here is a sample error message:
>     at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
>     at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:763)
>     at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3206)
>     at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:644)
>     at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)
>     at
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
>     at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1894)
>     at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1871)
>     at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
>     at
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
>     at
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)
>     at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:62)
>     at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
>     at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
>     at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
>     at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
>     at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
>     at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
>     at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
>     at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>     at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>     at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>     at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>     at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>     at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>     at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>     at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>     at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>     at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>     at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>     at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>     at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>     at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>     at org.eclipse.jetty.server.Server.handle(Server.java:534)
>     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>     at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>     at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>     at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>     at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
>     at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
>     at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
>     at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
>     at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
>     at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /local/var/solr/shard2/filesearch/data/index/_34w5.fdx: Too many open files
>     at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
>     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>     at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
>     at
> java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
>     at java.nio.file.Files.newOutputStream(Files.java:216)
>     at
> org.apache.lucene.store.FSDirectory$FSIndexOutput.&lt;init&gt;(FSDirectory.java:413)
>     at
> org.apache.lucene.store.FSDirectory$FSIndexOutput.&lt;init&gt;(FSDirectory.java:409)
>     at
> org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
>     at
> org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
>     at
> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
>     at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.&lt;init&gt;(CompressingStoredFieldsWriter.java:114)
>     at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
>     at
> org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
>     at
> org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:39)
>     at
> org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:46)
>     at
> org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:364)
>     at
> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:398)
>     at
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
>     at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
>     at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1571)
>     at
> org.apache.solr.update.DirectUpdateHandler2.updateDocument(DirectUpdateHandler2.java:924)
>     at
> org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:913)
>     at
> org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:302)
>     at
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
>     at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194)
>     at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
>     at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
>     at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1005)
>     at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
>     at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>     at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
>     at
> org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:205)
>     at
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261)
>     at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188)
>     at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
>     at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>     ... 33 more
>
>
>
> Thanks
> Nawab

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message