spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renato Falchi Brandão (JIRA) <>
Subject [jira] [Commented] (SPARK-15067) YARN executors are launched with fixed perm gen size
Date Tue, 03 May 2016 17:06:13 GMT


Renato Falchi Brandão commented on SPARK-15067:

Hi Devaraj,
I will create the PR, thank you!

> YARN executors are launched with fixed perm gen size
> ----------------------------------------------------
>                 Key: SPARK-15067
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.6.0, 1.6.1
>            Reporter: Renato Falchi Brandão
>            Priority: Minor
> It is impossible to change the executors max perm gen size using the property "spark.executor.extraJavaOptions"
when you are running on YARN.
> When the JVM option "-XX:MaxPermSize" is set through the property "spark.executor.extraJavaOptions",
Spark put it properly in the shell command that will start the JVM container but, in the ending
of command, it sets again this option using a fixed value of 256m, as you can see in the log
I've extracted:
> 2016-04-30 17:20:12 INFO  ExecutorRunnable:58 -
> ===============================================================================
> YARN executor launch context:
>   env:
>     CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>/usr/hdp/current/hadoop-client/*<CPS>/usr/hdp/current/hadoop-client/lib/*<CPS>/usr/hdp/current/hadoop-hdfs-client/*<CPS>/usr/hdp/current/hadoop-hdfs-client/lib/*<CPS>/usr/hdp/current/hadoop-yarn-client/*<CPS>/usr/hdp/current/hadoop-yarn-client/lib/*<CPS>/usr/hdp/mr-framework/hadoop/share/hadoop/mapreduce/*:/usr/hdp/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:/usr/hdp/mr-framework/hadoop/share/hadoop/common/*:/usr/hdp/mr-framework/hadoop/share/hadoop/common/lib/*:/usr/hdp/mr-framework/hadoop/share/hadoop/yarn/*:/usr/hdp/mr-framework/hadoop/share/hadoop/yarn/lib/*:/usr/hdp/mr-framework/hadoop/share/hadoop/hdfs/*:/usr/hdp/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/current/hadoop/lib/hadoop-lzo-0.6.0.jar:/etc/hadoop/conf/secure
>     SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1456962126505_329993
>     SPARK_YARN_CACHE_FILES_FILE_SIZES -> 191719054,166
>     SPARK_USER -> h_loadbd
>     SPARK_YARN_MODE -> true
>     SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1459806496093,1459808508343
>     SPARK_YARN_CACHE_FILES -> hdfs://xxxxx/user/datalab/hdp/spark/lib/spark-assembly-,hdfs://tlvcluster/user/datalab/hdp/spark/conf/hive-site.xml#hive-site.xml
>   command:
>     {{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms6144m -Xmx6144m
'-XX:+PrintGCDetails' '-XX:MaxPermSize=1024M' '-XX:+PrintGCTimeStamps'{{PWD}}/tmp
'-Dspark.akka.timeout=300000' '-Dspark.driver.port=62875' '-Dspark.rpc.askTimeout=300000'
'-Dspark.rpc.lookupTimeout=300000'<LOG_DIR> -XX:MaxPermSize=256m
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@
--executor-id 1 --hostname --cores 1 --app-id application_1456962126505_329993
--user-class-path file:$PWD/__app__.jar 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
> Analyzing the code is possible to see that all the options set in the property "spark.executor.extraJavaOptions"
are enclosed, one by one, in single quotes (ExecutorRunnable.scala:151) before the launcher
take the decision if a default value has to be provided or not for the option "-XX:MaxPermSize"
> This decision is taken examining all the options set and looking for a string starting
with the value "-XX:MaxPermSize" ( If that value is not found,
the default value is set.
> A string option starting without single quote will never be found, then, a default value
will always be provided.
> A possible solution is change the source code of in the line
> From-> if (arg.startsWith("-XX:MaxPermSize="))
> To-> if (arg.indexOf("-XX:MaxPermSize=") > -1)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message