spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anupama Joshi <anupama.jo...@gmail.com>
Subject Re: Spark UI and running spark-submit with --master yarn
Date Tue, 03 Mar 2015 00:14:31 GMT
Hi Marcelo,
Thanks for the quick reply.
I have a EMR cluster and I am running the spark-submit on the master node
in the cluster.
When I start the spark-submit , I see
15/03/02 23:48:33 INFO client.RMProxy: Connecting to ResourceManager at /
172.31.43.254:9022
But If I try that URL or the use the external DNS
ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9022
it does not work
What am I missing here ?
Thanks a lot for the help
-AJ


On Mon, Mar 2, 2015 at 3:50 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> What are you calling "<masternode>"? In yarn-cluster mode, the driver
> is running somewhere in your cluster, not on the machine where you run
> spark-submit.
>
> The easiest way to get to the Spark UI when using Yarn is to use the
> Yarn RM's web UI. That will give you a link to the application's UI
> regardless of whether it's running on client or cluster mode.
>
> On Mon, Mar 2, 2015 at 3:39 PM, Anupama Joshi <anupama.joshi@gmail.com>
> wrote:
> > Hi ,
> >
> >  When I run my application with --master yarn-cluster or --master yarn
> > --deploy-mode cluster , I can not  the spark UI at the  location --
> > <masternode>:4040Even if I am running the job , I can not see teh SPARK
> UI.
> > When I run with --master yarn --deploy-mode client  -- I see the Spark UI
> > but I cannot see my job  running.
> >
> > When I run spark-submit with --master local[*] , I see the spark UI , my
> job
> > everything (Thats great)
> >
> > Do I need to do some settings to see the UI?
> >
> > Thanks
> >
> > -AJ
> >
> >
> >
> >
> >
> >
>
>
>
> --
> Marcelo
>

Mime
View raw message