spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ankit tyagi <ankittyagi.mn...@gmail.com>
Subject OOM in spark driver
Date Wed, 02 Sep 2015 06:03:50 GMT
Hi All,

I am using spark-sql 1.3.1 with hadoop 2.4.0 version.  I am running sql
query against parquet files and wanted to save result on s3 but looks like
https://issues.apache.org/jira/browse/SPARK-2984 problem still coming while
saving data to s3.

Hence Now i am saving result on hdfs and with the help
of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil
in onApplicationEnd method. But  my job is getting failed with OOM in spark
driver.

*5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking each
executor to shut down*
*15/09/02 04:17:59 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor:
OutputCommitCoordinator stopped!*
*Exception in thread "Reporter" *
*Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "Reporter"*
*Exception in thread "SparkListenerBus" *
*Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "SparkListenerBus"*
*Exception in thread "Driver" *
*Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "Driver"*


Strage part is, result is getting saved on HDFS but while copying file job
is getting failed. size of file is under 1MB.

Any help or leads would be appreciated.

Mime
View raw message