spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grega Kešpret <gr...@celtra.com>
Subject Re: java.lang.OutOfMemoryError: Map failed
Date Wed, 25 Sep 2013 15:33:57 GMT
How do you determine that it legitimately runs out of memory? It does
"sc.textFile(s3://...)
-> filter -> map -> reduceByKey() -> saveAsTextFile(s3://)". The map
basically emits a key, value pair and the cardinality of key is <<< number
of original objects.

Grega
--
[image: Inline image 1]
*Grega Kešpret*
Analytics engineer

Celtra — Rich Media Mobile Advertising
celtra.com <http://www.celtra.com/> |
@celtramobile<http://www.twitter.com/celtramobile>


On Fri, Sep 20, 2013 at 11:55 PM, Reynold Xin <rxin@cs.berkeley.edu> wrote:

> It looks like it is actually legitimately running out of memory. What does
> application job do?
>
>
>
> --
> Reynold Xin, AMPLab, UC Berkeley
> http://rxin.org
>
>
>
> On Fri, Sep 20, 2013 at 11:57 AM, Reynold Xin <rxin@cs.berkeley.edu>wrote:
>
>>
>> Maybe you have a single key or a single partition that is extremely
>> large?  Can you try log the gc information to see what else is going on?
>>
>>
>>
>> --
>> Reynold Xin, AMPLab, UC Berkeley
>> http://rxin.org
>>
>>
>>
>> On Thu, Sep 19, 2013 at 8:43 AM, Grega Kešpret <grega@celtra.com> wrote:
>>
>>> Hi!
>>>
>>> I have a simple job, basically sc.textFile(s3://...) -> filter -> map ->
>>> reduceByKey() -> saveAsTextFile(s3://) that fails with OOM.
>>>
>>> Just before it fails I see a whole bunch of Full GC, each lasting 2-3
>>> secs that don't recover any substantial heap space. Then I see
>>>
>>> Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit
>>> Server VM warning: Attempt to deallocate stack guard pages failed.
>>> Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack
>>> guard pages failed.
>>>
>>> and finally
>>>
>>> 13/09/19 12:37:33 ERROR BlockManagerWorker: Exception handling buffer
>>> message
>>> java.io.IOException: Map failed
>>>     at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>>>     at spark.storage.DiskStore.getBytes(DiskStore.scala:86)
>>>     at spark.storage.BlockManager.getLocalBytes(BlockManager.scala:376)
>>>     at
>>> spark.storage.BlockManagerWorker.getBlock(BlockManagerWorker.scala:79)
>>>     at
>>> spark.storage.BlockManagerWorker.processBlockMessage(BlockManagerWorker.scala:58)
>>>     at
>>> spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:33)
>>>     at
>>> spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:33)
>>>     at
>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
>>>     at
>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
>>>     at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>>>     at
>>> scala.collection.IndexedSeqLike$Elements.foreach(IndexedSeqLike.scala:54)
>>>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
>>>     at
>>> spark.storage.BlockMessageArray.foreach(BlockMessageArray.scala:12)
>>>     at
>>> scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
>>>     at spark.storage.BlockMessageArray.map(BlockMessageArray.scala:12)
>>>     at
>>> spark.storage.BlockManagerWorker.onBlockMessageReceive(BlockManagerWorker.scala:33)
>>>     at
>>> spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:23)
>>>     at
>>> spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:23)
>>>     at
>>> spark.network.ConnectionManager.spark$network$ConnectionManager$$handleMessage(ConnectionManager.scala:269)
>>>     at
>>> spark.network.ConnectionManager$$anon$5.run(ConnectionManager.scala:235)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>     at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.lang.OutOfMemoryError: Map failed
>>>     at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>>     at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>>>     ... 22 more
>>>
>>> I am not caching anything, and I am already using quite a large number
>>> of numPartitions in reduceByKey (1000).
>>>
>>> The logs of one of the workers can be found here:
>>>
>>> https://dl.dropboxusercontent.com/u/921501/java.lang.OutOfMemoryError-Mapfailed.txt
>>>
>>> Grega
>>> --
>>> [image: Inline image 1]
>>> *Grega Kešpret*
>>> Analytics engineer
>>>
>>> Celtra — Rich Media Mobile Advertising
>>> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>>>
>>
>>
>

Mime
View raw message