lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralf Matulat <ralf.matu...@bundestag.de>
Subject Optimize fails with OutOfMemory Exception - sun.nio.ch.FileChannelImpl.map involved
Date Thu, 22 Sep 2011 09:27:38 GMT
Good morning!
Recently we slipped into an OOME by optimizing our index. It looks like 
it's regarding to the nio class and the memory-handling.
I'll try to describe the environment, the error and what we did to solve 
the problem. Nevertheless, none of our approaches was successful.

The environment:

- Tested with both, SOLR 3.3 & 3.4
- SuSE SLES 11 (X64)virtual machine with 16GB RAM
- ulimi: virtual memory 14834560 (14GB)
- Java: java-1_6_0-ibm-1.6.0-124.5
- Apache Tomcat/6.0.29

- Index Size (on filesystem): ~5GB, 1.1 million text documents.

The error:
First, building the index from scratch with a mysql DIH, with an empty 
index-Dir works fine.
Building an index with &command=full-import, when the old segment files 
still in place, fails with an OutOfMemoryException. Same as optimizing 
the index.
Doing an optimize fails after some time with:

SEVERE: java.io.IOException: background merge hit exception: 
_6p(3.4):Cv1150724 _70(3.4):Cv667 _73(3.4):Cv7 _72(3.4):Cv4 _71(3.4):Cv1 
into _74 [optimize]
         at 
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2552)
         at 
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2472)
         at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
         at 
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
         at 
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
         at 
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107)
         at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:61)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
         at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
         at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
         at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
         at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
         at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
         at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
         at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
         at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
         at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
         at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
         at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
         at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
         at java.lang.Thread.run(Thread.java:735)
Caused by: java.io.IOException: Map failed
         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:765)
         at 
org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264)
         at 
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
         at 
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:89)
         at 
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
         at 
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:710)
         at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4378)
         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3917)
         at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
         at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
Caused by: java.lang.OutOfMemoryError: Map failed
         at sun.nio.ch.FileChannelImpl.map0(Native Method)
         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:762)
         ... 9 more

Then we changed mergeScheduler and mergePolicy to

<mergeScheduler class="org.apache.lucene.index.SerialMergeScheduler" />
<mergePolicy class="org.apache.lucene.index.LogByteSizeMergePolicy"/>

which lead to a slightly different error-message:

SEVERE: java.io.IOException: Map failed
         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:765)
         at 
org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264)
         at 
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
         at 
org.apache.lucene.index.TermVectorsReader.<init>(TermVectorsReader.java:85)
         at 
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:221)
         at 
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
         at 
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:710)
         at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4248)
         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3917)
         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2725)
         at 
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2535)
         at 
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2472)
         at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
         at 
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
         at 
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
         at 
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107)
         at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:61)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
         at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
         at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
         at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
         at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
         at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
         at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
         at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
         at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
         at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
         at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
         at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
         at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
         at java.lang.Thread.run(Thread.java:735)
Caused by: java.lang.OutOfMemoryError: Map failed
         at sun.nio.ch.FileChannelImpl.map0(Native Method)
         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:762)
         ... 33 more

Both seem to have in common the "sun.nio.ch.FileChannelImpl.map" running 
out of memory.

We looked at "free" while doing the optimize and we saw, that the 
optimize ate up all the free RAM. When no RAM was left, the exception 
was thrown. No swapping etc. at all.


What we did to solve the problem:

We discovered a link from this list and tried the mentioned solutions:
http://www.mail-archive.com/solr-user@lucene.apache.org/msg54094.html

No success.
We set ulimit -v unlimited, changed mergePolicy (see above).
Last thing we tried: We decreased the RAM of the JVM. At least that 
worked. The optimize finished.

But on a long run this is no solution. Our index is growing constantly 
and RAM, which is sufficient today, maybe is not tomorrow. We wonder how 
the memory management works. If you are running another process on that 
machine, using a reasonable amount of RAM in a short period of time, an 
optimize would fail when the optimize starts at the wrong point in time.
Or what are you doing, if you got an 25GB index? Do you need a +50GB RAM 
Server to do an optimize? That does not make any sense and so there must 
be something we can do about it - something we missed.

Maybe someone get some bright ideas?
Thanks a lot and best regards
Ralf

Mime
View raw message