spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <>
Subject [jira] [Commented] (SPARK-1350) YARN ContainerLaunchContext should use cluster's JAVA_HOME
Date Mon, 31 Mar 2014 14:27:15 GMT


Thomas Graves commented on SPARK-1350:

I assume you are referring to the one in ClientBase?

It actually does not use the same java as specified by JAVA_HOME on the client.  If JAVA_HOME
is set on the client then it just adds $JAVA_HOME/bin to the java command to launch the AM.
Yarn can be configured (and is the default config) to put JAVA_HOME into the container launch
script, so the launch command will use the JAVA_HOME specified by yarn.   I agree its a bit
hacky and could be improved though.  On MapReduce it just always uses "$JAVA_HOME/bin/java"

The main idea is that I want $JAVA_HOME to be included in the launch command so that its not
relying on the default install of java on the system.  I want it to use the JAVA_HOME specified
by yarn and I also want to allow it to be overridden so that a user could choose to run either
32 bit or 64 bit jdk.  

export SPARK_YARN_USER_ENV="JAVA_HOME=/myyarinstall/jdk64/current"

I was told this goes against how spark is in general setup where it relies on whatever is
installed on the system so this was the compromise.

> YARN ContainerLaunchContext should use cluster's JAVA_HOME
> ----------------------------------------------------------
>                 Key: SPARK-1350
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 0.9.0
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>             Fix For: 1.0.0
> {code}
>     var javaCommand = "java"
>     val javaHome = System.getenv("JAVA_HOME")
>     if ((javaHome != null && !javaHome.isEmpty()) || env.isDefinedAt("JAVA_HOME"))
>       javaCommand = Environment.JAVA_HOME.$() + "/bin/java"
>     }
> {code}
> Currently, if JAVA_HOME is specified on the client, it will be used instead of the value
given on the cluster.  This makes it so that Java must be installed in the same place on the
client as on the cluster.
> This is a possibly incompatible change that we should get in before 1.0.

This message was sent by Atlassian JIRA

View raw message