Try setting SPARK_EXECUTOR_MEMORY=5g (not sure how many workers you are having), You can also set the executor memory while creating the sparkContext (like sparkContext.set("spark.executor.memory","5g") )

Thanks
Best Regards

On Mon, Oct 20, 2014 at 5:01 PM, Arian Pasquali <arian@arianpasquali.com> wrote:
Hi Akhil,
thanks for your help

but I was originally running without xmx option. With that I was just trying to push the limit of my heap size, but obviously doing it wrong.


2014-10-20 12:24 GMT+01:00 Akhil Das <akhil@sigmoidanalytics.com>:
Hi Arian,

You will get this exception because you are trying to create an array that is larger than the maximum contiguous block of memory in your Java VMs heap.

Here since you are setting Worker memory as 5Gb and you are exporting the *_OPTS as 8Gb, your application actually thinks it has 8Gb of memory where as it only has 5Gb and hence it exceeds the VM Limit.



Thanks
Best Regards

On Mon, Oct 20, 2014 at 4:42 PM, Arian Pasquali <arian@arianpasquali.com> wrote:
Hi, 
I’m using Spark 1.1.0 and I’m having some issues to setup memory options.
I get “Requested array size exceeds VM limit” and I’m probably missing something regarding memory configuration.

My server has 30G of memory and this are my current settings. 

##this one seams that was deprecated
export SPARK_MEM=‘25g’ 

## worker memory options seams to be the memory for each worker (by default we have a worker for each core)
export SPARK_WORKER_MEMORY=‘5g’ 

I probably need to specify some options using SPARK_DAEMON_JAVA_OPTS, but I’m not quite sure how.
I have tried some different options like the following, but I still couldn’t make it right:

export SPARK_DAEMON_JAVA_OPTS='-Xmx8G -XX:+UseCompressedOops'
export JAVA_OPTS='-Xmx8G -XX:+UseCompressedOops'

Does anyone has any idea how can I approach this?




14/10/11 13:00:16 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, targetRequestSize: 10066329
14/10/11 13:00:16 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 1566 non-empty blocks out of 1566 blocks
14/10/11 13:00:16 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote fetches in 4 ms
14/10/11 13:02:06 INFO ExternalAppendOnlyMap: Thread 63 spilling in-memory map of 3925 MB to disk (1 time so far)
14/10/11 13:05:17 INFO ExternalAppendOnlyMap: Thread 63 spilling in-memory map of 3925 MB to disk (2 times so far)
14/10/11 13:09:15 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 1566)
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Arrays.java:2271)
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
        at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876)
        at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1188)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
        at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
        at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
14/10/11 13:09:15 ERROR ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-2,5,main]
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Arrays.java:2271)
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140


Arian