spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bin Wang <binwang...@gmail.com>
Subject Missing Spark URL after staring the master
Date Mon, 03 Mar 2014 19:00:28 GMT
Hi there,

I have a CDH cluster set up, and I tried using the Spark parcel come with
Cloudera Manager, but it turned out they even don't have the run-example
shell command in the bin folder. Then I removed it from the cluster and
cloned the incubator-spark into the name node of my cluster, and built from
source there successfully with everything as default.

I ran a few examples and everything seems work fine in the local mode. Then
I am thinking about scale it to my cluster, which is what the "DISTRIBUTE +
ACTIVATE" command does in Cloudera Manager. I want to add all the datanodes
to the slaves and think I should run Spark in the standalone mode.

Say I am trying to set up Spark in the standalone mode following this
instruction:
https://spark.incubator.apache.org/docs/latest/spark-standalone.html
However, it says "Once started, the master will print out a
spark://HOST:PORT URL for itself, which you can use to connect workers to
it, or pass as the "master" argument to SparkContext. You can also find
this URL on the master's web UI, which is http://localhost:8080 by default."

After I started the master, there is no URL printed on the screen and
neither the web UI is running.
Here is the output:
[root@box incubator-spark]# ./sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to
/root/bwang_spark_new/incubator-spark/sbin/../logs/spark-root-org.apache.spark.deploy.master.Master-1-box.out

First Question: am I even in the ballpark to run Spark in standalone mode
if I try to fully utilize my cluster? I saw there are four ways to launch
Spark on a cluster, AWS-EC2, Spark in standalone, Apache Meso, Hadoop
Yarn... which I guess standalone mode is the way to go?

Second Question: how to get the Spark URL of the cluster, why the output is
not like what the instruction says?

Best regards,

Bin

Mime
View raw message