spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Spark UI and running spark-submit with --master yarn
Date Tue, 03 Mar 2015 00:23:40 GMT
Default RM Web UI port is 8088 (configurable
through yarn.resourcemanager.webapp.address)

Cheers

On Mon, Mar 2, 2015 at 4:14 PM, Anupama Joshi <anupama.joshi@gmail.com>
wrote:

> Hi Marcelo,
> Thanks for the quick reply.
> I have a EMR cluster and I am running the spark-submit on the master node
> in the cluster.
> When I start the spark-submit , I see
> 15/03/02 23:48:33 INFO client.RMProxy: Connecting to ResourceManager at /
> 172.31.43.254:9022
> But If I try that URL or the use the external DNS
> ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9022
> it does not work
> What am I missing here ?
> Thanks a lot for the help
> -AJ
>
>
> On Mon, Mar 2, 2015 at 3:50 PM, Marcelo Vanzin <vanzin@cloudera.com>
> wrote:
>
>> What are you calling "<masternode>"? In yarn-cluster mode, the driver
>> is running somewhere in your cluster, not on the machine where you run
>> spark-submit.
>>
>> The easiest way to get to the Spark UI when using Yarn is to use the
>> Yarn RM's web UI. That will give you a link to the application's UI
>> regardless of whether it's running on client or cluster mode.
>>
>> On Mon, Mar 2, 2015 at 3:39 PM, Anupama Joshi <anupama.joshi@gmail.com>
>> wrote:
>> > Hi ,
>> >
>> >  When I run my application with --master yarn-cluster or --master yarn
>> > --deploy-mode cluster , I can not  the spark UI at the  location --
>> > <masternode>:4040Even if I am running the job , I can not see teh SPARK
>> UI.
>> > When I run with --master yarn --deploy-mode client  -- I see the Spark
>> UI
>> > but I cannot see my job  running.
>> >
>> > When I run spark-submit with --master local[*] , I see the spark UI ,
>> my job
>> > everything (Thats great)
>> >
>> > Do I need to do some settings to see the UI?
>> >
>> > Thanks
>> >
>> > -AJ
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Marcelo
>>
>
>

Mime
View raw message