spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Spark Bug? job fails to run when given options on spark-submit (but starts and fails without)
Date Wed, 22 Oct 2014 16:25:12 GMT
Hi Michael Campbell,

Are you deploying against yarn or standalone mode? In yarn try setting the
shell variables SPARK_EXECUTOR_MEMORY=2G in standalone try and
set SPARK_WORKER_MEMORY=2G.


Cheers,

Holden :)

On Thu, Oct 16, 2014 at 2:22 PM, Michael Campbell <
michael.campbell@gmail.com> wrote:

> TL;DR - a spark SQL job fails with an OOM (Out of heap space) error.  If
> given "--executor-memory" values, it won't even start.  Even (!) if the
> values given ARE THE SAME AS THE DEFAULT.
>
>
>
> Without --executor-memory:
>
> 14/10/16 17:14:58 INFO TaskSetManager: Serialized task 1.0:64 as 14710
> bytes in 1 ms
> 14/10/16 17:14:58 WARN TaskSetManager: Lost TID 26 (task 1.0:25)
> 14/10/16 17:14:58 WARN TaskSetManager: Loss was due to
> java.lang.OutOfMemoryError
> java.lang.OutOfMemoryError: Java heap space
>         at
> parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:609)
>         at
> parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:360)
> ...
>
>
> USING --executor-memory (WITH ANY VALUE), even "1G" which is the default:
>
> Parsed arguments:
>   master                  spark://<redacted>:7077
>   deployMode              null
>   executorMemory          1G
> ...
>
> System properties:
> spark.executor.memory -> 1G
> spark.eventLog.enabled -> true
> ...
>
> 14/10/16 17:14:23 INFO TaskSchedulerImpl: Adding task set 1.0 with 678
> tasks
> 14/10/16 17:14:38 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
>
>
> Spark 1.0.0.  Is this a bug?
>
>


-- 
Cell : 425-233-8271

Mime
View raw message