spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Lee <alee...@hotmail.com>
Subject RE: Is spark 1.0.0 "spark-shell --master=yarn" running in yarn-cluster mode or yarn-client mode?
Date Wed, 21 May 2014 22:57:46 GMT
Ah, forgot the -verbose option. Thanks Andrew. That is very helpful. 

Date: Wed, 21 May 2014 11:07:55 -0700
Subject: Re: Is spark 1.0.0 "spark-shell --master=yarn" running in yarn-cluster mode or yarn-client
mode?
From: andrew@databricks.com
To: user@spark.apache.org

The answer is actually yarn-client. A quick way to find out:
$ bin/spark-shell --master yarn --verbose
>From the system properties you can see spark.master is set to "yarn-client." From the
code, this is because args.deployMode is null, and so it's not equal to "cluster" and so it
falls into the second "if" case you mentioned:

if (args.deployMode != "cluster" && args.master.startsWith("yarn")) {
  args.master = "yarn-client"}

2014-05-21 10:57 GMT-07:00 Andrew Lee <alee526@hotmail.com>:




Does anyone know if:
./bin/spark-shell --master yarn 
is running yarn-cluster or yarn-client by default?

Base on source code:







./core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala


    if (args.deployMode == "cluster" && args.master.startsWith("yarn")) {

      args.master = "yarn-cluster"
    }
    if (args.deployMode != "cluster" && args.master.startsWith("yarn")) {

      args.master = "yarn-client"














    }

It looks like the answer is yarn-cluster mode.
I want to confirm this with the community, thanks. 		 	   		  


 		 	   		  
Mime
View raw message