From user-return-27995-apmail-spark-user-archive=spark.apache.org@spark.apache.org Tue Mar 3 22:19:28 2015 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7BCB217328 for ; Tue, 3 Mar 2015 22:19:28 +0000 (UTC) Received: (qmail 45225 invoked by uid 500); 3 Mar 2015 22:19:24 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 45150 invoked by uid 500); 3 Mar 2015 22:19:24 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 45139 invoked by uid 99); 3 Mar 2015 22:19:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Mar 2015 22:19:24 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of roni.epi112@gmail.com designates 209.85.192.41 as permitted sender) Received: from [209.85.192.41] (HELO mail-qg0-f41.google.com) (209.85.192.41) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Mar 2015 22:18:58 +0000 Received: by mail-qg0-f41.google.com with SMTP id f51so10434464qge.0 for ; Tue, 03 Mar 2015 14:17:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=YxnLeUhGolzLnD6ZNxL7LRyFeKX1LhPQBnOAVEkuCKM=; b=ia83yUWElGMKC3EgIAZ1bFXvg9RXslRdYu/D2j9Sn9MTmpxnT03PvHAuYX5/k5Toyq dC4n3jEOJX4yhNKjFwyh2QNzIpV8hMZzHlDS/wE771e+sO13CVQUpiPhQ4WQ1MzqzBxs jVspM8FNvdA+Rx29b3pM11HgTakQLP8CG/ME30fYqqLZam0UBU8iz10d/3sBFWil2gXe q2O4ODmg1PXWLx+Phg5CForbSDXbM7gHG4r1fH6B0ZMKTcwgQOsKvnWah2RLWCTznryg E9QgcKqcHGDQsOrXOLT6TFXhcfN6fiQuMzLswzT3W5/xESglPDu9T01EoA1uQylDXNv+ hq9g== MIME-Version: 1.0 X-Received: by 10.55.54.71 with SMTP id d68mr2043771qka.42.1425421046790; Tue, 03 Mar 2015 14:17:26 -0800 (PST) Received: by 10.140.27.200 with HTTP; Tue, 3 Mar 2015 14:17:26 -0800 (PST) In-Reply-To: References: Date: Tue, 3 Mar 2015 14:17:26 -0800 Message-ID: Subject: Re: Resource manager UI for Spark applications From: roni To: Zhan Zhang Cc: Ted Yu , "user@spark.apache.org" Content-Type: multipart/alternative; boundary=001a11470f422644fe051069b3e4 X-Virus-Checked: Checked by ClamAV on apache.org --001a11470f422644fe051069b3e4 Content-Type: text/plain; charset=UTF-8 ah!! I think I know what you mean. My job was just in "accepted" stage for a long time as it was running a huge file. But now that it is in running stage , I can see it . I can see it at post 9046 though instead of 4040 . But I can see it. Thanks -roni On Tue, Mar 3, 2015 at 1:19 PM, Zhan Zhang wrote: > In Yarn (Cluster or client), you can access the spark ui when the app is > running. After app is done, you can still access it, but need some extra > setup for history server. > > Thanks. > > Zhan Zhang > > On Mar 3, 2015, at 10:08 AM, Ted Yu wrote: > > bq. changing the address with internal to the external one , but still > does not work. > Not sure what happened. > For the time being, you can use yarn command line to pull container log > (put in your appId and container Id): > yarn logs -applicationId application_1386639398517_0007 -containerId > container_1386639398517_0007_01_000019 > > Cheers > > On Tue, Mar 3, 2015 at 9:50 AM, roni wrote: > >> Hi Ted, >> I used s3://support.elasticmapreduce/spark/install-spark to install >> spark on my EMR cluster. It is 1.2.0. >> When I click on the link for history or logs it takes me to >> >> http://ip-172-31-43-116.us-west-2.compute.internal:9035/node/containerlogs/container_1424105590052_0070_01_000001/hadoop >> and I get - >> >> The server at *ip-172-31-43-116.us >> -west-2.compute.internal* can't be found, >> because the DNS lookup failed. DNS is the network service that translates a >> website's name to its Internet address. This error is most often caused by >> having no connection to the Internet or a misconfigured network. It can >> also be caused by an unresponsive DNS server or a firewall preventing Google >> Chrome from accessing the network. >> I tried changing the address with internal to the external one , but >> still does not work. >> Thanks >> _roni >> >> >> On Tue, Mar 3, 2015 at 9:05 AM, Ted Yu wrote: >> >>> bq. spark UI does not work for Yarn-cluster. >>> >>> Can you be a bit more specific on the error(s) you saw ? >>> >>> What Spark release are you using ? >>> >>> Cheers >>> >>> On Tue, Mar 3, 2015 at 8:53 AM, Rohini joshi >>> wrote: >>> >>>> Sorry , for half email - here it is again in full >>>> Hi , >>>> I have 2 questions - >>>> >>>> 1. I was trying to use Resource Manager UI for my SPARK application >>>> using yarn cluster mode as I observed that spark UI does not work for >>>> Yarn-cluster. >>>> IS that correct or am I missing some setup? >>>> >>>> 2. when I click on Application Monitoring or history , i get >>>> re-directed to some linked with internal Ip address. Even if I replace that >>>> address with the public IP , it still does not work. What kind of setup >>>> changes are needed for that? >>>> >>>> Thanks >>>> -roni >>>> >>>> On Tue, Mar 3, 2015 at 8:45 AM, Rohini joshi >>>> wrote: >>>> >>>>> Hi , >>>>> I have 2 questions - >>>>> >>>>> 1. I was trying to use Resource Manager UI for my SPARK application >>>>> using yarn cluster mode as I observed that spark UI does not work for >>>>> Yarn-cluster. >>>>> IS that correct or am I missing some setup? >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>> >> > > --001a11470f422644fe051069b3e4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
ah!! I think I know what you mean. My job w= as just in "accepted" stage for a long time as it was running a h= uge file.
But now that it is in running stage , I can see it . I = can see it at post 9046 though instead of 4040 . But=C2=A0 I can see it.
Thanks
-roni

On Tue, Mar 3, 2015 at 1:19 PM, Zhan Zhang <zz= hang@hortonworks.com> wrote:
In Yarn (Cluster or client), you can access the spark ui when the app is ru= nning. After app is done, you can still access it, but need some extra setu= p for history server.

Thanks.

Zhan Zhang

On Mar 3, 2015, at 10:08 AM, Ted Yu <yuzhihong@gmail.com> wrote:

bq.=C2=A0changing the address with internal to the ex= ternal one , but still does not work.=C2=A0
= Not sure what happened.
= For the time being, you can use yarn comma= nd line to pull container log (put in your appId and container Id):
yarn logs -applicationId application_1386639398517_0007 -containerId c= ontainer_1386639398517_0007_01_000019

Cheers

On Tue, Mar 3, 2015 at 9:50 AM, roni <ron= i.epi112@gmail.com> wrote:
Hi Ted,
=C2=A0I=C2=A0 used s3://support.elasticmapreduce/spark/install-spark to install spark o= n my EMR cluster. It is 1.2.0.
=C2=A0When I click on the link for history or logs it takes me to
http://ip-172-31-43-116.us-west-2.compute.internal:9035/node/containerlogs= /container_1424105590052_0070_01_000001/hadoop
=C2=A0and I get -

The server at=C2=A0ip-172-31-43-116.us-west-2.compute.internal=C2=A0can't be found, because the DNS lookup failed. DNS is the network service that t= ranslates a website's name to its Internet address. This error is most = often caused by having no connection to the Internet or a misconfigured net= work. It can also be caused by an unresponsive DNS server or a firewall preventing=C2=A0Goog= le Chrome=C2=A0from accessing the network.
I tried=C2=A0 changing the address with internal to the external one , but s= till does not work.
Thanks
_roni


On Tue, Mar 3, 2015 at 9:05 AM, Ted Yu <yuzhi= hong@gmail.com> wrote:
bq.=C2=A0spar= k UI does not work for Yarn-cluster.

Can you be a bit more spe= cific on the error(s) you saw ?

What Spark release are yo= u using ?

Cheers

On Tue, Mar 3, 2015 at 8:53 AM, Rohini joshi <roni.epi112@= gmail.com> wrote:
Sorry , for half email - here it is again in full
Hi ,
I have 2 questions -

=C2=A01. I was trying to use Resource Manager UI for my SPARK application u= sing yarn cluster mode as I observed that spark UI does not work for Yarn-c= luster.
IS that correct or am I missing some setup?

2. when I click on Application Monitoring or history , i get re-directed to= some linked with internal Ip address. Even if I replace that address with = the public IP , it still does not work.=C2=A0 What kind of setup changes ar= e needed for that?

Thanks
-roni

On Tue, Mar 3, 2015 at 8:45 AM, Rohini joshi <roni.epi112@= gmail.com> wrote:
Hi ,
I have 2 questions -

=C2=A01. I was trying to use Resource Manager UI for my SPARK application u= sing yarn cluster mode as I observed that spark UI does not work for Yarn-c= luster.
IS that correct or am I missing some setup?










--001a11470f422644fe051069b3e4--