lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doron Cohen (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (LUCENE-488) adding docs with large (binary) fields of 5mb causes OOM regardless of heap size
Date Mon, 31 Dec 2007 10:57:43 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Doron Cohen resolved LUCENE-488.
--------------------------------

    Resolution: Fixed

This problem was resolved by LUCENE-843, after which stored fields are written directly into
the directory (therefore not consuming aggregated RAM). 

It is interesting that the test provided here was allocating a new byte buffer of 2 - 10 MB
for each added doc. This by itself couldeventually lead to OOMs because as the program ran
longer it was becoming harder to alocate consecutive chunks of those sizes.  Enhancing binary
fields with offset and length (?)  would allow applications to reuse the input byte array
and allocate less of those.

> adding docs with large (binary) fields of 5mb causes OOM regardless of heap size
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-488
>                 URL: https://issues.apache.org/jira/browse/LUCENE-488
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 1.9
>         Environment: Linux asimov 2.6.6.hoss1 #1 SMP Tue Jul 6 16:31:01 PDT 2004 i686
GNU/Linux
>            Reporter: Hoss Man
>         Attachments: TestBigBinary.java
>
>
> as reported by George Washington in a message to java-user@lucene.apache.org with subect
"Storing large text or binary source documents in the index and memory usage" arround 2006-01-21
there seems to be a problem with adding docs containing really large fields.
> I'll attach a test case in a moment, note that (for me) regardless of how big i make
my heap size, and regardless of what value I set  MIN_MB to, once it starts trying to make
documents of containing 5mb of data, it can only add 9 before it rolls over and dies.
> here's the output from the code as i will attach in a moment...
>     [junit] Testsuite: org.apache.lucene.document.TestBigBinary
>     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 78.656 sec
>     [junit] ------------- Standard Output ---------------
>     [junit] NOTE: directory will not be cleaned up automatically...
>     [junit] Dir: /tmp/org.apache.lucene.document.TestBigBinary.97856146.100iters.4mb
>     [junit] iters completed: 100
>     [junit] totalBytes Allocated: 419430400
>     [junit] NOTE: directory will not be cleaned up automatically...
>     [junit] Dir: /tmp/org.apache.lucene.document.TestBigBinary.97856146.100iters.5mb
>     [junit] iters completed: 9
>     [junit] totalBytes Allocated: 52428800
>     [junit] ------------- ---------------- ---------------
>     [junit] Testcase: testBigBinaryFields(org.apache.lucene.document.TestBigBinary):
   Caused an ERROR
>     [junit] Java heap space
>     [junit] java.lang.OutOfMemoryError: Java heap space
>     [junit] Test org.apache.lucene.document.TestBigBinary FAILED

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message