spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Date Mon, 20 Oct 2014 11:42:29 GMT
Try setting SPARK_EXECUTOR_MEMORY=5g (not sure how many workers you are
having), You can also set the executor memory while creating the
sparkContext (like *sparkContext.set("spark.executor.memory","5g")* )

Thanks
Best Regards

On Mon, Oct 20, 2014 at 5:01 PM, Arian Pasquali <arian@arianpasquali.com>
wrote:

> Hi Akhil,
> thanks for your help
>
> but I was originally running without xmx option. With that I was just
> trying to push the limit of my heap size, but obviously doing it wrong.
>
>
>
>
> Arian Pasquali
> http://about.me/arianpasquali
>
> 2014-10-20 12:24 GMT+01:00 Akhil Das <akhil@sigmoidanalytics.com>:
>
>> Hi Arian,
>>
>> You will get this exception because you are trying to create an array
>> that is larger than the maximum contiguous block of memory in your Java VMs
>> heap.
>>
>> Here since you are setting Worker memory as *5Gb* and you are exporting
>> the *_OPTS as *8Gb*, your application actually thinks it has 8Gb of
>> memory where as it only has 5Gb and hence it exceeds the VM Limit.
>>
>>
>>
>> Thanks
>> Best Regards
>>
>> On Mon, Oct 20, 2014 at 4:42 PM, Arian Pasquali <arian@arianpasquali.com>
>> wrote:
>>
>>> Hi,
>>> I’m using Spark 1.1.0 and I’m having some issues to setup memory options.
>>> I get “Requested array size exceeds VM limit” and I’m probably missing
>>> something regarding memory configuration
>>> <https://spark.apache.org/docs/1.1.0/configuration.html>.
>>>
>>> My server has 30G of memory and this are my current settings.
>>>
>>> ##this one seams that was deprecated
>>> export SPARK_MEM=‘25g’
>>>
>>> ## worker memory options seams to be the memory for each worker (by
>>> default we have a worker for each core)
>>> export SPARK_WORKER_MEMORY=‘5g’
>>>
>>> I probably need to specify some options using SPARK_DAEMON_JAVA_OPTS,
>>> but I’m not quite sure how.
>>> I have tried some different options like the following, but I still
>>> couldn’t make it right:
>>>
>>> export SPARK_DAEMON_JAVA_OPTS='-Xmx8G -XX:+UseCompressedOops'
>>> export JAVA_OPTS='-Xmx8G -XX:+UseCompressedOops'
>>>
>>> Does anyone has any idea how can I approach this?
>>>
>>>
>>>
>>>
>>> 14/10/11 13:00:16 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>>> maxBytesInFlight: 50331648, targetRequestSize: 10066329
>>> 14/10/11 13:00:16 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>>> Getting 1566 non-empty blocks out of 1566 blocks
>>> 14/10/11 13:00:16 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>>> Started 0 remote fetches in 4 ms
>>> 14/10/11 13:02:06 INFO ExternalAppendOnlyMap: Thread 63 spilling
>>> in-memory map of 3925 MB to disk (1 time so far)
>>> 14/10/11 13:05:17 INFO ExternalAppendOnlyMap: Thread 63 spilling
>>> in-memory map of 3925 MB to disk (2 times so far)
>>> 14/10/11 13:09:15 ERROR Executor: Exception in task 0.0 in stage 0.0
>>> (TID 1566)
>>> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>>>         at java.util.Arrays.copyOf(Arrays.java:2271)
>>>         at
>>> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
>>>         at
>>> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
>>>         at
>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
>>>         at
>>> java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876)
>>>         at
>>> java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785)
>>>         at
>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1188)
>>>         at
>>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
>>>         at
>>> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
>>>         at
>>> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
>>>         at
>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>         at java.lang.Thread.run(Thread.java:745)
>>> 14/10/11 13:09:15 ERROR ExecutorUncaughtExceptionHandler: Uncaught
>>> exception in thread Thread[Executor task launch worker-2,5,main]
>>> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>>>         at java.util.Arrays.copyOf(Arrays.java:2271)
>>>         at
>>> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
>>>         at
>>> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
>>>         at
>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140
>>>
>>>
>>> Arian
>>>
>>
>>
>

Mime
View raw message