phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Praveen Murugesan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PHOENIX-990) OOM caused by order by query returning all rows
Date Wed, 18 Jun 2014 22:58:24 GMT

     [ https://issues.apache.org/jira/browse/PHOENIX-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Praveen Murugesan updated PHOENIX-990:
--------------------------------------

    Attachment: PHOENIX-990.patch

Increasing the MappedByteBuffer default mapping size to 128KB from 1KB. 

The OOMs were caused by byte buffer size being really small, this involves allocation about
20K 1KB buffers, which are un referenced immediately. Since GC doesn't clear up direct buffers
quickly, we run into a OOM. Increasing the default buffer size relieve some of the GC pressure
and doesn't OOM for thresholds I tested.

I will attach the test i used in a separate patch for reference.

> OOM caused by order by query returning all rows
> -----------------------------------------------
>
>                 Key: PHOENIX-990
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-990
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 3.0.0, 4.0.0
>            Reporter: Mujtaba Chohan
>            Assignee: Praveen Murugesan
>         Attachments: PHOENIX-990.patch
>
>
> OOM error with the following stack trace with large number of rows for query with order
by returning all rows without limit or aggregation (ex. select * from table order by col1).
Created a local perf. test to verify it when this gets fixed. Also performance.py script can
be used to generate a few mil. rows.
> Originally reported by @zenmehra.
> Stack:
> ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.DoNotRetryIOException: PERFORMANCE_5000000,,1400524730456.c62cccdac8cffd098d236f5e282564bb.:
java.io.IOException: Map failed
> 	at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:73)
> 	at org.apache.phoenix.coprocessor.ScanRegionObserver.getTopNScanner(ScanRegionObserver.java:221)
> 	at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:197)
> 	at org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:66)
> 	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1316)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.internalOpenScanner(HRegionServer.java:2573)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2541)
> 	at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323)
> 	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> Caused by: java.lang.RuntimeException: java.io.IOException: Map failed
> 	at org.apache.phoenix.iterate.MappedByteBufferSortedQueue.offer(MappedByteBufferSortedQueue.java:73)
> 	at org.apache.phoenix.iterate.MappedByteBufferSortedQueue.offer(MappedByteBufferSortedQueue.java:43)
> 	at java.util.AbstractQueue.add(AbstractQueue.java:68)
> 	at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:224)
> 	at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:169)
> 	at org.apache.phoenix.coprocessor.ScanRegionObserver.getTopNScanner(ScanRegionObserver.java:216)
> 	... 10 more
> Caused by: java.io.IOException: Map failed
> 	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> 	at org.apache.phoenix.iterate.MappedByteBufferSortedQueue$MappedByteBufferPriorityQueue.writeResult(MappedByteBufferSortedQueue.java:299)
> 	at org.apache.phoenix.iterate.MappedByteBufferSortedQueue.offer(MappedByteBufferSortedQueue.java:65)
> 	... 15 more
> Caused by: java.lang.OutOfMemoryError: Map failed
> 	at sun.nio.ch.FileChannelImpl.map0(Native Method)
> 	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message