spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <>
Subject [jira] [Commented] (SPARK-1350) YARN ContainerLaunchContext should use cluster's JAVA_HOME
Date Mon, 31 Mar 2014 17:41:16 GMT


Sandy Ryza commented on SPARK-1350:

bq. It actually does not use the same java as specified by JAVA_HOME on the client.
Oops, you're right.  I was misunderstanding the code.  This is less of a big deal than I thought.

bq. The main idea is that I want $JAVA_HOME to be included in the launch command so that its
not relying on the default install of java on the system.
I 100% agree that this is the right thing to do.  What still seems a little weird to me is
that the use of JAVA_HOME on the cluster depends on whether the client JAVA_HOME is set.

bq. I was told this goes against how spark is in general setup where it relies on whatever
is installed on the system so this was the compromise.
So this has already been discussed and always using JAVA_HOME in the launch command (never
just "java") was rejected for this reason?  If all YARN apps accessed java in the same way,
the way that MapReduce does, I think it would make things a lot simpler operationally.

> YARN ContainerLaunchContext should use cluster's JAVA_HOME
> ----------------------------------------------------------
>                 Key: SPARK-1350
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 0.9.0
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>             Fix For: 1.0.0
> {code}
>     var javaCommand = "java"
>     val javaHome = System.getenv("JAVA_HOME")
>     if ((javaHome != null && !javaHome.isEmpty()) || env.isDefinedAt("JAVA_HOME"))
>       javaCommand = Environment.JAVA_HOME.$() + "/bin/java"
>     }
> {code}
> Currently, if JAVA_HOME is specified on the client, it will be used instead of the value
given on the cluster.  This makes it so that Java must be installed in the same place on the
client as on the cluster.
> This is a possibly incompatible change that we should get in before 1.0.

This message was sent by Atlassian JIRA

View raw message