db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mike matrigali <mikema...@gmail.com>
Subject Re: OutOfMemoryError
Date Thu, 04 Jun 2015 21:52:00 GMT
Did you get the error when adding the primary key?

If you got the error on a different query, could you post that query, and possibly
the query plan being executed?  A possible workaround might be to see if you
could create a different index such that the system could pick a query plan where
the sort is not needed.

As you mention derby does try to estimate ahead of time amount of memory, but it
has a number of problems:
1) the java interfaces do not work well for estimating amount of memory available in
      the future.
2) derby estimates for blob/clob size are often wrong.  I believe xml uses clob implementation
      internally.  I would guess the technical problem here is that derby underestimated the
sort
      size.

Providing a reproducible test case that you can check into JIRA will increase the likelyhood
that
a developer will look at the issue.

/mikem

On 6/3/2015 11:39 PM, Mattias Andersson wrote:
>
> We got a 'java.lang.OutOfMemoryError: Java heap space' when using a Derby disk database
with the following schema.
>
>
> CREATE TABLE TEST (ID BIGINT NOT NULL
>
>
> ,NEXT _ID BIGINT
>
> ,PREV_ ID BIGINT
>
> ,TEST_XML XML);
>
> ALTER TABLE TABLE
>
> ADD PRIMARY KEY
>
> (ID);
>
>
> I opened the heap dump with Eclipse Memory Analyzer and found that tree instances of

> org.apache.derby.impl.store.access.sort.MergeInserter occupies 290 MB of memory (that
is 66 % of the total heap)
>
>
> The problem seems to be that MergeInserter holds a sort buffer that contains complete
rows and theTEST_XML is about 160 kB each. 
> MergeInserter has an advanced algorithm to calculate the allowed size of the buffer,
but is it necessary store the complete row in 
> the buffer, i.e. the non-indexed values (TEST_XML is this case)?
>
>
> Is there some property I can set to limit the buffer size or is it possible to rewrite
the SQL schema someway to exclude the XML 
> type from the buffer?
>
>
> Br Mattias
>
>
>
>
> Following is the dominator tree from Eclipse Memory Analyzer:
>
>
>
> org.apache.derby.impl.store.access.sort.MergeInserter @ 0xe3f72e20|72 |127 533 080 |30,76%
>
> |- org.apache.derby.impl.store.access.sort.SortBuffer @ 0xe3f73018|48 |127 532 792 |30,76%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed5e0|40 |164 384 |0,04%
>
> ||'- org.apache.derby.iapi.types.DataValueDescriptor[4] @ 0xee3da280|32 |164 344 |0,04%
>
> |||- org.apache.derby.iapi.types.SQLClob @ 0xee3da2a0|56 |164 240 |0,04%
>
> ||||- java.lang.String @ 0xee3da2d8< ?xml version="1.0"?...|24 |164 160 |0,04%
>
> ||||- char[1][] @ 0xee3da2f0|24 |24 |0,00%
>
> |||'- Total: 2 entries|||
>
> |||- org.apache.derby.iapi.types.SQLLongint @ 0xee3da308|24 |24 |0,00%
>
> |||- org.apache.derby.iapi.types.SQLLongint @ 0xee3da320|24 |24 |0,00%
>
> |||- org.apache.derby.iapi.types.SQLLongint @ 0xee3da338|24 |24 |0,00%
>
> ||'- Total: 4 entries|||
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed608|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed630|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed658|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed680|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed6a8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed6d0|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed6f8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed720|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedd50|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedd78|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedda0|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3feddc8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedff8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fef5c8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fef5f0|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fef618|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff01f8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff04c8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0608|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0630|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0658|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0680|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff06a8|40 |164 384 |0,04%
>
> | |- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff06d0|40 |164 384 |0,04%
>
> |'- Total: 25 of 780 entries; 755
>


-- 
email:    Mike Matrigali - mikemapp1@gmail.com
linkedin: https://www.linkedin.com/in/MikeMatrigali


Mime
View raw message