lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <j...@apache.org>
Subject [jira] [Created] (LUCENE-6383) MemoryPostings fst encoding can be surprisingly inefficient (especially in tests, with payloads)
Date Wed, 01 Apr 2015 04:29:53 GMT
Robert Muir created LUCENE-6383:
-----------------------------------

             Summary: MemoryPostings fst encoding can be surprisingly inefficient (especially
in tests, with payloads)
                 Key: LUCENE-6383
                 URL: https://issues.apache.org/jira/browse/LUCENE-6383
             Project: Lucene - Core
          Issue Type: Bug
            Reporter: Robert Muir


I just worked around this in 2 nightly OOM fails.

One was TestDuelingCodecs, the other was TestIndexWriterForceMerge's space usage test.

In general the trend is the same, it seems the more documents you merge, you just get bigger
and bigger FST outputs and the size of this PF in ram and on disk grows in a way you don't
expect. E.g. merging 300KB of segments resulted in 450KB single segment, and memory usage
gets absurdly high.

The issue seems especially aggravated in tests, when MockAnalyzer adds lots of payloads.

Maybe it should encode the postings data in a more efficient way? Can it just use a Long output
pointing into a RAMFile or something? Or maybe there is just a crazy bug?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message