spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Davidson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
Date Fri, 01 Jul 2016 16:26:11 GMT

    [ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359215#comment-15359215
] 

Andrew Davidson commented on SPARK-15829:
-----------------------------------------

Hi Sean

I am not sure how to check the value of 'SPARK_MASTER_HOST'. I looked at the documentation
pags http://spark.apache.org/docs/latest/configuration.html and  http://spark.apache.org/docs/latest/ec2-scripts.html.
They do not mention SPARK_MASTER_HOST

when I submit my jobs I use 
MASTER_URL=spark://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:6066

I use the stand alone cluster manager

I think the problem may be that the web UI assume the driver is always running on the master
machine. I assume the cluster manager decides which worker the driver will run on. Is there
a way for the web UI to discover where the driver is running?

On my master 
[ec2-user@ip-172-31-22-140 conf]$ pwd
/root/spark/conf
[ec2-user@ip-172-31-22-140 conf]$ cat slaves
ec2-54-193-94-207.us-west-1.compute.amazonaws.com
ec2-54-67-13-246.us-west-1.compute.amazonaws.com
ec2-54-67-48-49.us-west-1.compute.amazonaws.com
ec2-54-193-104-169.us-west-1.compute.amazonaws.com
[ec2-user@ip-172-31-22-140 conf]$ 


[ec2-user@ip-172-31-22-140 conf]$ grep SPARK_MASTER_HOST *
[ec2-user@ip-172-31-22-140 conf]$ pwd
/root/spark/conf
[ec2-user@ip-172-31-22-140 conf]$ grep SPARK_MASTER_HOST *
[ec2-user@ip-172-31-22-140 conf]$ 

ec2-user@ip-172-31-22-140 sbin]$ pwd
/root/spark/sbin
[ec2-user@ip-172-31-22-140 sbin]$ grep SPARK_MASTER_HOST *

[ec2-user@ip-172-31-22-140 bin]$ !grep
grep SPARK_MASTER_HOST *
[ec2-user@ip-172-31-22-140 bin]$ 

Thanks for looking into this

Andy


> spark master webpage links to application UI broke when running in cluster mode
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-15829
>                 URL: https://issues.apache.org/jira/browse/SPARK-15829
>             Project: Spark
>          Issue Type: Bug
>          Components: EC2
>    Affects Versions: 1.6.1
>         Environment: AWS ec2 cluster
>            Reporter: Andrew Davidson
>            Priority: Critical
>
> Hi 
> I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2
> I use the stand alone cluster manager. I have a streaming app running in cluster mode.
I notice the master webpage links to the application UI page are incorrect
> It does not look like jira will let my upload images. I'll try and describe the web pages
and the bug
> My master is running on
> http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:8080/
> It has a section marked "applications". If I click on one of the running application
ids I am taken to a page showing "Executor Summary".  This page has a link to teh 'application
detail UI'  the url is http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:4041/
> Notice it things the application UI is running on the cluster master.
> It is actually running on the same machine as the driver on port 4041. I was able to
reverse engine the url by noticing the private ip address is part of the worker id . For example
	worker-20160322041632-172.31.23.201-34909
> next I went on the aws ec2 console to find the public DNS name for this machine 
> http://ec2-54-193-104-169.us-west-1.compute.amazonaws.com:4041/streaming/
> Kind regards
> Andy



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message