lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralf Matulat <ralf.matu...@bundestag.de>
Subject Re: Optimize fails with OutOfMemory Exception - sun.nio.ch.FileChannelImpl.map involved
Date Thu, 22 Sep 2011 13:00:12 GMT
Dear Mike,
thanks for your your reply.
Just a couple of minutes we found a solution or - to be honest - where 
we went wrong.
Our failure was the use of ulimit. We missed, that ulimit sets the vmem 
for each shell seperatly. So we set 'ulimit -v unlimited' on a shell, 
thinking that we've done the job correctly.
As we recognized our mistake, we added 'ulimit -v unlimited' to our  
init-Skript of the tomcat-instance and now it looks like everything 
works as aspected.
Need some further testing with the java versions, but I'm quite optimistic.
Best regards
Ralf

Am 22.09.2011 14:46, schrieb Michael McCandless:
> Are you sure you are using a 64 bit JVM?
>
> Are you sure you really changed your vmem limit to unlimited?  That
> should have resolved the OOME from mmap.
>
> Or: can you run "cat /proc/sys/vm/max_map_count"?  This is a limit on
> the total number of maps in a single process, that Linux imposes.  But
> the default limit is usually high (64K), so it'd be surprising if you
> are hitting that unless it's lower in your env.
>
> The amount of [free] RAM on the machine should have no bearing on
> whether mmap succeeds or fails; it's the available address space (32
> bit is tiny; 64 bit is immense) and then any OS limits imposed.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Thu, Sep 22, 2011 at 5:27 AM, Ralf Matulat<ralf.matulat@bundestag.de>  wrote:
>> Good morning!
>> Recently we slipped into an OOME by optimizing our index. It looks like it's
>> regarding to the nio class and the memory-handling.
>> I'll try to describe the environment, the error and what we did to solve the
>> problem. Nevertheless, none of our approaches was successful.
>>
>> The environment:
>>
>> - Tested with both, SOLR 3.3&  3.4
>> - SuSE SLES 11 (X64)virtual machine with 16GB RAM
>> - ulimi: virtual memory 14834560 (14GB)
>> - Java: java-1_6_0-ibm-1.6.0-124.5
>> - Apache Tomcat/6.0.29
>>
>> - Index Size (on filesystem): ~5GB, 1.1 million text documents.
>>
>> The error:
>> First, building the index from scratch with a mysql DIH, with an empty
>> index-Dir works fine.
>> Building an index with&command=full-import, when the old segment files
>> still in place, fails with an OutOfMemoryException. Same as optimizing the
>> index.
>> Doing an optimize fails after some time with:
>>
>> SEVERE: java.io.IOException: background merge hit exception:
>> _6p(3.4):Cv1150724 _70(3.4):Cv667 _73(3.4):Cv7 _72(3.4):Cv4 _71(3.4):Cv1
>> into _74 [optimize]
>>         at
>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2552)
>>         at
>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2472)
>>         at
>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
>>         at
>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>>         at
>> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
>>         at
>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107)
>>         at
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:61)
>>         at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>         at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>         at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>         at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>         at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>         at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>         at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>>         at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>>         at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
>>         at
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>         at java.lang.Thread.run(Thread.java:735)
>> Caused by: java.io.IOException: Map failed
>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:765)
>>         at
>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264)
>>         at
>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
>>         at
>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:89)
>>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
>>         at
>> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:710)
>>         at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4378)
>>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3917)
>>         at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
>>         at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
>> Caused by: java.lang.OutOfMemoryError: Map failed
>>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:762)
>>         ... 9 more
>>
>> Then we changed mergeScheduler and mergePolicy to
>>
>> <mergeScheduler class="org.apache.lucene.index.SerialMergeScheduler" />
>> <mergePolicy class="org.apache.lucene.index.LogByteSizeMergePolicy"/>
>>
>> which lead to a slightly different error-message:
>>
>> SEVERE: java.io.IOException: Map failed
>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:765)
>>         at
>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264)
>>         at
>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
>>         at
>> org.apache.lucene.index.TermVectorsReader.<init>(TermVectorsReader.java:85)
>>         at
>> org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:221)
>>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
>>         at
>> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:710)
>>         at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4248)
>>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3917)
>>         at
>> org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
>>         at
>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2725)
>>         at
>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2535)
>>         at
>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2472)
>>         at
>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
>>         at
>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>>         at
>> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
>>         at
>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107)
>>         at
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:61)
>>         at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>         at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>         at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>         at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>         at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>         at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>         at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>>         at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>>         at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
>>         at
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>         at java.lang.Thread.run(Thread.java:735)
>> Caused by: java.lang.OutOfMemoryError: Map failed
>>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:762)
>>         ... 33 more
>>
>> Both seem to have in common the "sun.nio.ch.FileChannelImpl.map" running out
>> of memory.
>>
>> We looked at "free" while doing the optimize and we saw, that the optimize
>> ate up all the free RAM. When no RAM was left, the exception was thrown. No
>> swapping etc. at all.
>>
>>
>> What we did to solve the problem:
>>
>> We discovered a link from this list and tried the mentioned solutions:
>> http://www.mail-archive.com/solr-user@lucene.apache.org/msg54094.html
>>
>> No success.
>> We set ulimit -v unlimited, changed mergePolicy (see above).
>> Last thing we tried: We decreased the RAM of the JVM. At least that worked.
>> The optimize finished.
>>
>> But on a long run this is no solution. Our index is growing constantly and
>> RAM, which is sufficient today, maybe is not tomorrow. We wonder how the
>> memory management works. If you are running another process on that machine,
>> using a reasonable amount of RAM in a short period of time, an optimize
>> would fail when the optimize starts at the wrong point in time.
>> Or what are you doing, if you got an 25GB index? Do you need a +50GB RAM
>> Server to do an optimize? That does not make any sense and so there must be
>> something we can do about it - something we missed.
>>
>> Maybe someone get some bright ideas?
>> Thanks a lot and best regards
>> Ralf
>>


-- 
Ralf Matulat
Deutscher Bundestag
Platz der Republik 1
11011 Berlin
Referat IT 1 - Anwendungsadministration
ralf.matulat@bundestag.de
Tel.: 030 - 227 34260


Mime
View raw message