spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: --driver-memory parameter doesn't work for spark-submmit on yarn?
Date Wed, 01 Apr 2015 06:40:22 GMT
Once you submit the job do a ps aux | grep spark-submit and see how much is
the heap space allocated to the process (the -Xmx params), if you are
seeing a lower value you could try increasing it yourself with:

export _JAVA_OPTIONS="-Xmx5g"

Thanks
Best Regards

On Wed, Apr 1, 2015 at 1:57 AM, Shuai Zheng <szheng.code@gmail.com> wrote:

> Hi All,
>
>
>
> Below is the my shell script:
>
>
>
> /home/hadoop/spark/bin/spark-submit --driver-memory=5G
> --executor-memory=40G --master yarn-client --class
> com.***.FinancialEngineExecutor /home/hadoop/lib/my.jar
> s3://bucket/vriscBatchConf.properties
>
>
>
> My driver will load some resources and then broadcast to all executors.
>
>
>
> That resources is only 600MB in ser format, but I always has out of memory
> exception, it looks like the driver doesn’t allocate right memory to my
> driver.
>
>
>
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>
>         at java.lang.reflect.Array.newArray(Native Method)
>
>         at java.lang.reflect.Array.newInstance(Array.java:70)
>
>         at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1670)
>
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344)
>
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>
>         at
> com.***.executor.support.S3FileUtils.loadCache(S3FileUtils.java:68)
>
>
>
> Do I do anything wrong here?
>
>
>
> And no matter how much I set for --driver-memory value (from 512M to
> 20G), it always give me error on the same line (that line try to load a
> 600MB java serialization file). So looks like the script doesn’t allocate
> right memory to driver in my case?
>
>
>
> Regards,
>
>
>
> Shuai
>

Mime
View raw message