lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rich Cariens <richcari...@gmail.com>
Subject Re: MMapDirectory failed to map a 23G compound index segment
Date Mon, 12 Sep 2011 14:17:06 GMT
Thanks. It's definitely repeatable and I may spend some time plumbing this
further. I'll let the list know if I find anything.

The problem went away once I optimized the index down to a single segment
using a simple IndexWriter driver. This was a bit strange since the
resulting index contained similarly large (> 23G) files. The JVM didn't seem
to have any trouble MMap'ing those.

No, I don't need (or necessarily want) to use compound index file formats.
That was actually a goof on my part which I've since corrected :).

On Fri, Sep 9, 2011 at 9:42 PM, Lance Norskog <goksron@gmail.com> wrote:

> I remember now: by memory-mapping one block of address space that big, the
> garbage collector has problems working around it. If the OOM is repeatable,
> you could try watching the app with jconsole and watch the memory spaces.
>
> Lance
>
> On Thu, Sep 8, 2011 at 8:58 PM, Lance Norskog <goksron@gmail.com> wrote:
>
> > Do you need to use the compound format?
> >
> > On Thu, Sep 8, 2011 at 3:57 PM, Rich Cariens <richcariens@gmail.com
> >wrote:
> >
> >> I should add some more context:
> >>
> >>   1. the problem index included several cfs segment files that were
> around
> >>   4.7G, and
> >>   2. I'm running four SOLR instances on the same box, all of which have
> >>   similiar problem indeces.
> >>
> >> A colleague thought perhaps I was bumping up against my 256,000 open
> files
> >> ulimit. Do the MultiMMapIndexInput ByteBuffer arrays each consume a file
> >> handle/descriptor?
> >>
> >> On Thu, Sep 8, 2011 at 5:19 PM, Rich Cariens <richcariens@gmail.com>
> >> wrote:
> >>
> >> > FWiW I optimized the index down to a single segment and now I have no
> >> > trouble opening an MMapDirectory on that index, even though the 23G
> cfx
> >> > segment file remains.
> >> >
> >> >
> >> > On Thu, Sep 8, 2011 at 4:27 PM, Rich Cariens <richcariens@gmail.com
> >> >wrote:
> >> >
> >> >> Thanks for the response. "free -g" reports:
> >> >>
> >> >>         total        used        free        shared        buffers
> >> >> cached
> >> >> Mem:      141          95          46             0
> >> >> 0            93
> >> >> -/+ buffers/cache:      2         139
> >> >> Swap:       3           0           3
> >> >>
> >> >> 2011/9/7 Fran├žois Schiettecatte <fschiettecatte@gmail.com>
> >> >>
> >> >>> My memory of this is a little rusty but isn't mmap also limited
by
> mem
> >> +
> >> >>> swap on the box? What does 'free -g' report?
> >> >>>
> >> >>> Fran├žois
> >> >>>
> >> >>> On Sep 7, 2011, at 12:25 PM, Rich Cariens wrote:
> >> >>>
> >> >>> > Ahoy ahoy!
> >> >>> >
> >> >>> > I've run into the dreaded OOM error with MMapDirectory on
a 23G
> cfs
> >> >>> compound
> >> >>> > index segment file. The stack trace looks pretty much like
every
> >> other
> >> >>> trace
> >> >>> > I've found when searching for OOM & "map failed"[1]. My
> >> configuration
> >> >>> > follows:
> >> >>> >
> >> >>> > Solr 1.4.1/Lucene 2.9.3 (plus
> >> >>> > SOLR-1969<https://issues.apache.org/jira/browse/SOLR-1969>
> >> >>> > )
> >> >>> > CentOS 4.9 (Final)
> >> >>> > Linux 2.6.9-100.ELsmp x86_64 yada yada yada
> >> >>> > Java SE (build 1.6.0_21-b06)
> >> >>> > Hotspot 64-bit Server VM (build 17.0-b16, mixed mode)
> >> >>> > ulimits:
> >> >>> >    core file size     (blocks, -c)     0
> >> >>> >    data seg size    (kbytes, -d)     unlimited
> >> >>> >    file size     (blocks, -f)     unlimited
> >> >>> >    pending signals    (-i)     1024
> >> >>> >    max locked memory     (kbytes, -l)     32
> >> >>> >    max memory size     (kbytes, -m)     unlimited
> >> >>> >    open files    (-n)     256000
> >> >>> >    pipe size     (512 bytes, -p)     8
> >> >>> >    POSIX message queues     (bytes, -q)     819200
> >> >>> >    stack size    (kbytes, -s)     10240
> >> >>> >    cpu time    (seconds, -t)     unlimited
> >> >>> >    max user processes     (-u)     1064959
> >> >>> >    virtual memory    (kbytes, -v)     unlimited
> >> >>> >    file locks    (-x)     unlimited
> >> >>> >
> >> >>> > Any suggestions?
> >> >>> >
> >> >>> > Thanks in advance,
> >> >>> > Rich
> >> >>> >
> >> >>> > [1]
> >> >>> > ...
> >> >>> > java.io.IOException: Map failed
> >> >>> > at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> >> >>> > at
> >> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(Unknown
> >> >>> > Source)
> >> >>> > at
> >> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(Unknown
> >> >>> > Source)
> >> >>> > at org.apache.lucene.store.MMapDirectory.openInput(Unknown
Source)
> >> >>> > at
> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(Unknown
> >> >>> Source)
> >> >>> >
> >> >>> > at org.apache.lucene.index.SegmentReader.get(Unknown Source)
> >> >>> > at org.apache.lucene.index.SegmentReader.get(Unknown Source)
> >> >>> > at org.apache.lucene.index.DirectoryReader.<init>(Unknown
Source)
> >> >>> > at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(Unknown
> >> >>> Source)
> >> >>> > at org.apache.lucene.index.DirectoryReader$1.doBody(Unknown
> Source)
> >> >>> > at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Unknown
> >> >>> > Source)
> >> >>> > at org.apache.lucene.index.DirectoryReader.open(Unknown Source)
> >> >>> > at org.apache.lucene.index.IndexReader.open(Unknown Source)
> >> >>> > ...
> >> >>> > Caused by: java.lang.OutOfMemoryError: Map failed
> >> >>> > at sun.nio.ch.FileChannelImpl.map0(Native Method)
> >> >>> > ...
> >> >>>
> >> >>>
> >> >>
> >> >
> >>
> >
> >
> >
> > --
> > Lance Norskog
> > goksron@gmail.com
> >
> >
>
>
> --
> Lance Norskog
> goksron@gmail.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message