spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From CodingCat <>
Subject [GitHub] incubator-spark pull request: [SPARK-1092] remove SPARK_MEM usage ...
Date Sat, 15 Feb 2014 00:12:03 GMT
GitHub user CodingCat reopened a pull request:

    [SPARK-1092] remove SPARK_MEM usage in sparkcontext.scala
    Currently, users will usually set SPARK_MEM to control the memory usage of driver programs,
(in spark-class)
    92 JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$SPARK_LIBRARY_PATH"
    if they didn't set spark.executor.memory, the value in this environment variable will
also affect the memory usage of executors, because the following lines in SparkContext
    privatespark val executorMemory = conf.getOption("spark.executor.memory")
    since SPARK_MEM has been (proposed to) deprecated in SPARK-929 (
and the corresponding PR (
    we should remove this line

You can merge this pull request into a Git repository by running:

    $ git pull clean_spark_mem

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #602
commit 31ea1cfe4231c7ff14f2721fa2be99baa43c29d0
Author: CodingCat <>
Date:   2014-02-14T23:31:03Z

    remove SPARK_MEM usage in sparkcontext.scala

commit c9b4872d366a5d41400eacd8d2feef0020d0c109
Author: CodingCat <>
Date:   2014-02-15T00:14:25Z

    print warning information if user use SPARK_MEM to regulate executor memory usage


View raw message