spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: OOM in spark driver
Date Fri, 04 Sep 2015 09:27:21 GMT
Or you can increase the driver heap space (export _JAVA_OPTIONS="-Xmx5g")

Thanks
Best Regards

On Wed, Sep 2, 2015 at 11:57 PM, Mike Hynes <91mbbh@gmail.com> wrote:

> Just a thought; this has worked for me before on standalone client
> with a similar OOM error in a driver thread. Try setting:
> export SPARK_DAEMON_MEMORY=4G #or whatever size you can afford on your
> machine
> in your environment/spark-env.sh before running spark-submit.
> Mike
>
> On 9/2/15, ankit tyagi <ankittyagi.mnnit@gmail.com> wrote:
> > Hi All,
> >
> > I am using spark-sql 1.3.1 with hadoop 2.4.0 version.  I am running sql
> > query against parquet files and wanted to save result on s3 but looks
> like
> > https://issues.apache.org/jira/browse/SPARK-2984 problem still coming
> while
> > saving data to s3.
> >
> > Hence Now i am saving result on hdfs and with the help
> > of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil
> > in onApplicationEnd method. But  my job is getting failed with OOM in
> spark
> > driver.
> >
> > *5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking each
> > executor to shut down*
> > *15/09/02 04:17:59 INFO
> > scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor:
> > OutputCommitCoordinator stopped!*
> > *Exception in thread "Reporter" *
> > *Exception: java.lang.OutOfMemoryError thrown from the
> > UncaughtExceptionHandler in thread "Reporter"*
> > *Exception in thread "SparkListenerBus" *
> > *Exception: java.lang.OutOfMemoryError thrown from the
> > UncaughtExceptionHandler in thread "SparkListenerBus"*
> > *Exception in thread "Driver" *
> > *Exception: java.lang.OutOfMemoryError thrown from the
> > UncaughtExceptionHandler in thread "Driver"*
> >
> >
> > Strage part is, result is getting saved on HDFS but while copying file
> job
> > is getting failed. size of file is under 1MB.
> >
> > Any help or leads would be appreciated.
> >
>
>
> --
> Thanks,
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message