spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhan Zhang <zzh...@hortonworks.com>
Subject Re: Resource manager UI for Spark applications
Date Tue, 03 Mar 2015 21:19:37 GMT
In Yarn (Cluster or client), you can access the spark ui when the app is running. After app
is done, you can still access it, but need some extra setup for history server.

Thanks.

Zhan Zhang

On Mar 3, 2015, at 10:08 AM, Ted Yu <yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>>
wrote:

bq. changing the address with internal to the external one , but still does not work.
Not sure what happened.
For the time being, you can use yarn command line to pull container log (put in your appId
and container Id):
yarn logs -applicationId application_1386639398517_0007 -containerId container_1386639398517_0007_01_000019

Cheers

On Tue, Mar 3, 2015 at 9:50 AM, roni <roni.epi112@gmail.com<mailto:roni.epi112@gmail.com>>
wrote:
Hi Ted,
 I  used s3://support.elasticmapreduce/spark/install-spark to install spark on my EMR cluster.
It is 1.2.0.
 When I click on the link for history or logs it takes me to
http://ip-172-31-43-116.us-west-2.compute.internal:9035/node/containerlogs/container_1424105590052_0070_01_000001/hadoop
 and I get -

The server at ip-172-31-43-116.us<http://ip-172-31-43-116.us>-west-2.compute.internal
can't be found, because the DNS lookup failed. DNS is the network service that translates
a website's name to its Internet address. This error is most often caused by having no connection
to the Internet or a misconfigured network. It can also be caused by an unresponsive DNS server
or a firewall preventing Google Chrome from accessing the network.
I tried  changing the address with internal to the external one , but still does not work.
Thanks
_roni


On Tue, Mar 3, 2015 at 9:05 AM, Ted Yu <yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>>
wrote:
bq. spark UI does not work for Yarn-cluster.

Can you be a bit more specific on the error(s) you saw ?

What Spark release are you using ?

Cheers

On Tue, Mar 3, 2015 at 8:53 AM, Rohini joshi <roni.epi112@gmail.com<mailto:roni.epi112@gmail.com>>
wrote:
Sorry , for half email - here it is again in full
Hi ,
I have 2 questions -

 1. I was trying to use Resource Manager UI for my SPARK application using yarn cluster mode
as I observed that spark UI does not work for Yarn-cluster.
IS that correct or am I missing some setup?

2. when I click on Application Monitoring or history , i get re-directed to some linked with
internal Ip address. Even if I replace that address with the public IP , it still does not
work.  What kind of setup changes are needed for that?

Thanks
-roni

On Tue, Mar 3, 2015 at 8:45 AM, Rohini joshi <roni.epi112@gmail.com<mailto:roni.epi112@gmail.com>>
wrote:
Hi ,
I have 2 questions -

 1. I was trying to use Resource Manager UI for my SPARK application using yarn cluster mode
as I observed that spark UI does not work for Yarn-cluster.
IS that correct or am I missing some setup?










Mime
View raw message