spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: How to set JAVA_HOME with --deploy-mode cluster
Date Mon, 27 Oct 2014 14:33:23 GMT
You can do *which java* command and see where exactly is your java being
installed and export it as the JAVA_HOME.

[image: Inline image 1]

Thanks
Best Regards

On Mon, Oct 27, 2014 at 7:56 PM, Thomas Risberg <trisberg@pivotal.io> wrote:

> Hi,
>
> I'm trying to run the SparkPi example using `--deploy-mode cluster`. Not
> having much luck. Looks like the JAVA_HOME on my Mac is used for launching
> the app on the Spark cluster running on CentOS. How can I set the JAVA_HOME
> or better yet, how can I configure Spark to use the JAVA_HOME configured
> for the cluster?
>
> Here is the sysout from launching the job:
>
> $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi
> --deploy-mode cluster --master spark://borneo:7077
> hdfs://borneo:8020/xd/spark/spark-examples_2.10-1.2.0-SNAPSHOT.jar 100
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 14/10/27 10:19:13 INFO SecurityManager: Changing view acls to: trisberg,
> 14/10/27 10:19:13 INFO SecurityManager: Changing modify acls to: trisberg,
> 14/10/27 10:19:13 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(trisberg, );
> users with modify permissions: Set(trisberg, )
> 14/10/27 10:19:13 INFO Slf4jLogger: Slf4jLogger started
> 14/10/27 10:19:14 INFO Utils: Successfully started service 'driverClient'
> on port 49999.
> Sending launch command to spark://borneo:7077
> Driver successfully submitted as driver-20141027091913-0007
> ... waiting before polling master for driver state
> ... polling master for driver state
> State of driver-20141027091913-0007 is ERROR
> Exception from cluster was: java.io.IOException: Cannot run program
> "/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/bin/java"
> (in directory
> "/home/hadoop/spark-1.1.0-bin-hadoop2.4/work/driver-20141027091913-0007"):
> error=2, No such file or directory
> java.io.IOException: Cannot run program
> "/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/bin/java"
> (in directory
> "/home/hadoop/spark-1.1.0-bin-hadoop2.4/work/driver-20141027091913-0007"):
> error=2, No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
> at
> org.apache.spark.deploy.worker.ProcessBuilderLike$$anon$4.start(DriverRunner.scala:238)
> at
> org.apache.spark.deploy.worker.DriverRunner.runCommandWithRetry(DriverRunner.scala:200)
> at org.apache.spark.deploy.worker.DriverRunner.org
> $apache$spark$deploy$worker$DriverRunner$$launchDriver(DriverRunner.scala:183)
> at
> org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:88)
> Caused by: java.io.IOException: error=2, No such file or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
> at java.lang.ProcessImpl.start(ProcessImpl.java:130)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
> ... 4 more
>
> Thanks,
> Thomas
>

Mime
View raw message