spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marius Soutier <mps....@gmail.com>
Subject Re: Spark Web UI port 4040 not working
Date Wed, 27 Jul 2016 19:06:08 GMT
That's to be expected - the application UI is not started by the master, but by the driver.
So the UI will run on the machine that submits the job.


> On 26.07.2016, at 15:49, Jestin Ma <jestinwith.an.e@gmail.com> wrote:
> 
> I did netstat -apn | grep 4040 on machine 6, and I see
> 
> tcp        0      0 :::4040                     :::*                        LISTEN  
   30597/java
> 
> What does this mean?
> 
> On Tue, Jul 26, 2016 at 6:47 AM, Jestin Ma <jestinwith.an.e@gmail.com <mailto:jestinwith.an.e@gmail.com>>
wrote:
> I do not deploy using cluster mode and I don't use EC2.
> 
> I just read that launching as client mode: "the driver is launched directly within the
spark-submit process which acts as a client to the cluster."
> 
> My current setup is that I have cluster machines 1, 2, 3, 4, 5, with 1 being the master.

> I submit from another cluster machine 6 in client mode. So I'm taking that the driver
is launched in my machine 6.
> 
> On Tue, Jul 26, 2016 at 6:38 AM, Jacek Laskowski <jacek@japila.pl <mailto:jacek@japila.pl>>
wrote:
> Hi,
> 
> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
> figure out where the driver runs and use the machine's IP.
> 
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/ <https://medium.com/@jaceklaskowski/>
> Mastering Apache Spark http://bit.ly/mastering-apache-spark <http://bit.ly/mastering-apache-spark>
> Follow me at https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski>
> 
> 
> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma <jestinwith.an.e@gmail.com <mailto:jestinwith.an.e@gmail.com>>
wrote:
> > I tried doing that on my master node.
> > I got nothing.
> > However, I grep'd port 8080 and I got the standalone UI.
> >
> > On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <giaosudau@gmail.com <mailto:giaosudau@gmail.com>>
wrote:
> >>
> >> You’re running in StandAlone Mode?
> >> Usually inside active task it will show the address of current job.
> >> or you can check in master node by using netstat -apn | grep 4040
> >>
> >>
> >>
> >> > On Jul 26, 2016, at 8:21 AM, Jestin Ma <jestinwith.an.e@gmail.com <mailto:jestinwith.an.e@gmail.com>>
> >> > wrote:
> >> >
> >> > Hello, when running spark jobs, I can access the master UI (port 8080
> >> > one) no problem. However, I'm confused as to how to access the web UI to
see
> >> > jobs/tasks/stages/etc.
> >> >
> >> > I can access the master UI at http://<master-node>:8080. But port
4040
> >> > gives me a -connection cannot be reached-.
> >> >
> >> > Is the web UI http://<master-node> with a port of 4040?
> >> >
> >> > I'm running my Spark job on a cluster machine and submitting it to a
> >> > master node part of the cluster. I heard of ssh tunneling; is that relevant
> >> > here?
> >> >
> >> > Thank you!
> >>
> >
> 
> 


Mime
View raw message