spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <and...@databricks.com>
Subject Re: [VOTE] Release Apache Spark 1.6.0 (RC3)
Date Thu, 17 Dec 2015 20:14:03 GMT
@syepes

I just run Spark 1.6 (881f254) on YARN with Hadoop 2.4.0. I was able to run
a simple application in cluster mode successfully.

Can you verify whether the org.apache.spark.yarn.ApplicationMaster class
exists in your assembly jar?

jar -tf assembly.jar | grep ApplicationMaster

-Andrew


2015-12-17 7:44 GMT-08:00 syepes <syepes@gmail.com>:

> -1 (YARN Cluster deployment mode not working)
>
> I have just tested 1.6 (d509194b) on our HDP 2.3 platform and the cluster
> mode does not seem work. It looks like some parameter are not being passed
> correctly.
> This example works correctly with 1.5.
>
> # spark-submit --master yarn --deploy-mode cluster --num-executors 1
> --properties-file $PWD/spark-props.conf --class
> org.apache.spark.examples.SparkPi
> /opt/spark/lib/spark-examples-1.6.0-SNAPSHOT-hadoop2.7.1.jar
>
> Error: Could not find or load main class
> org.apache.spark.deploy.yarn.ApplicationMaster
>
> spark-props.conf
> -----------------------------
> spark.driver.extraJavaOptions                -Dhdp.version=2.3.2.0-2950
> spark.driver.extraLibraryPath
>
> /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
> spark.executor.extraJavaOptions              -Dhdp.version=2.3.2.0-2950
> spark.executor.extraLibraryPath
>
> /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
> -----------------------------
>
> I will try to do some more debugging on this issue.
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-6-0-RC3-tp15660p15692.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message