spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Davidson <A...@SantaCruzIntegration.com>
Subject Re: Spark Web UI port 4040 not working
Date Tue, 26 Jul 2016 17:16:38 GMT
Yup in cluster mode you need to figure out what machine the driver is
running on. That is the machine the UI will run on

https://issues.apache.org/jira/browse/SPARK-15829

You may also have fire wall issues

Here are some notes I made about how to figure out what machine the driver
is running on when using cluster mode

Application UI Bug work around
If our application is running in cluster mode the Master Console UI will
incorrectly assume the Application UI¹s are running on the master machine.
The UI is actually running on the same machine as the driver
To find the URL to the apps running in cluster mode
1. find the worker id for the app you are interested in.
* on the master console UI go to the Running App¹s section
* you should see a column ŒWorker¹
* a worker is something like worker-20160322041632-172.31.23.201-34909
2. got to the AWS ec2 console
3. from the private ip find the public DNS name
* the private ip in our example is 172.31.23.201

From:  Jacek Laskowski <jacek@japila.pl>
Date:  Tuesday, July 26, 2016 at 6:38 AM
To:  Jestin Ma <jestinwith.an.e@gmail.com>
Cc:  Chanh Le <giaosudau@gmail.com>, "user @spark" <user@spark.apache.org>
Subject:  Re: Spark Web UI port 4040 not working

> Hi,
> 
> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
> figure out where the driver runs and use the machine's IP.
> 
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
> 
> 
> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma <jestinwith.an.e@gmail.com> wrote:
>>  I tried doing that on my master node.
>>  I got nothing.
>>  However, I grep'd port 8080 and I got the standalone UI.
>> 
>>  On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <giaosudau@gmail.com> wrote:
>>> 
>>>  You¹re running in StandAlone Mode?
>>>  Usually inside active task it will show the address of current job.
>>>  or you can check in master node by using netstat -apn | grep 4040
>>> 
>>> 
>>> 
>>>>  > On Jul 26, 2016, at 8:21 AM, Jestin Ma <jestinwith.an.e@gmail.com>
>>>>  > wrote:
>>>>  >
>>>>  > Hello, when running spark jobs, I can access the master UI (port 8080
>>>>  > one) no problem. However, I'm confused as to how to access the web
UI to
>>>> see
>>>>  > jobs/tasks/stages/etc.
>>>>  >
>>>>  > I can access the master UI at http://<master-node>:8080. But
port 4040
>>>>  > gives me a -connection cannot be reached-.
>>>>  >
>>>>  > Is the web UI http://<master-node> with a port of 4040?
>>>>  >
>>>>  > I'm running my Spark job on a cluster machine and submitting it to
a
>>>>  > master node part of the cluster. I heard of ssh tunneling; is that
>>>> relevant
>>>>  > here?
>>>>  >
>>>>  > Thank you!
>>> 
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> 
> 



Mime
View raw message