spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Davidson (JIRA)" <>
Subject [jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
Date Fri, 01 Jul 2016 16:26:11 GMT


Andrew Davidson commented on SPARK-15829:

Hi Sean

I am not sure how to check the value of 'SPARK_MASTER_HOST'. I looked at the documentation
pags and
They do not mention SPARK_MASTER_HOST

when I submit my jobs I use 

I use the stand alone cluster manager

I think the problem may be that the web UI assume the driver is always running on the master
machine. I assume the cluster manager decides which worker the driver will run on. Is there
a way for the web UI to discover where the driver is running?

On my master 
[ec2-user@ip-172-31-22-140 conf]$ pwd
[ec2-user@ip-172-31-22-140 conf]$ cat slaves
[ec2-user@ip-172-31-22-140 conf]$ 

[ec2-user@ip-172-31-22-140 conf]$ grep SPARK_MASTER_HOST *
[ec2-user@ip-172-31-22-140 conf]$ pwd
[ec2-user@ip-172-31-22-140 conf]$ grep SPARK_MASTER_HOST *
[ec2-user@ip-172-31-22-140 conf]$ 

ec2-user@ip-172-31-22-140 sbin]$ pwd
[ec2-user@ip-172-31-22-140 sbin]$ grep SPARK_MASTER_HOST *

[ec2-user@ip-172-31-22-140 bin]$ !grep
[ec2-user@ip-172-31-22-140 bin]$ 

Thanks for looking into this


> spark master webpage links to application UI broke when running in cluster mode
> -------------------------------------------------------------------------------
>                 Key: SPARK-15829
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: EC2
>    Affects Versions: 1.6.1
>         Environment: AWS ec2 cluster
>            Reporter: Andrew Davidson
>            Priority: Critical
> Hi 
> I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2
> I use the stand alone cluster manager. I have a streaming app running in cluster mode.
I notice the master webpage links to the application UI page are incorrect
> It does not look like jira will let my upload images. I'll try and describe the web pages
and the bug
> My master is running on
> It has a section marked "applications". If I click on one of the running application
ids I am taken to a page showing "Executor Summary".  This page has a link to teh 'application
detail UI'  the url is
> Notice it things the application UI is running on the cluster master.
> It is actually running on the same machine as the driver on port 4041. I was able to
reverse engine the url by noticing the private ip address is part of the worker id . For example
> next I went on the aws ec2 console to find the public DNS name for this machine 
> Kind regards
> Andy

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message