spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <and...@databricks.com>
Subject Re: Web UI
Date Fri, 05 Sep 2014 18:07:32 GMT
Sure, you can request it by filing an issue here:
https://issues.apache.org/jira/browse/SPARK


2014-09-05 6:50 GMT-07:00 Ruebenacker, Oliver A <
Oliver.Ruebenacker@altisource.com>:

>
>
>      Hello,
>
>
>
>   Thanks for the explanation. So events are stored internally as JSON, but
> there is no official support for having Spark serve that JSON via HTTP? So
> if I wanted to write an app that monitors Spark, I would either have to
> scrape the web UI in HTML or rely on unofficial JSON features? That is
> quite surprising, because I would expect dumping out the JSON would be
> easier for Spark developers to implement than converting it to HTML.
>
>
>
>   Do I get that right? Should I make a feature request? Thanks!
>
>
>
>      Best, Oliver
>
>
>
> *From:* Andrew Or [mailto:andrew@databricks.com]
> *Sent:* Thursday, September 04, 2014 2:11 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* Akhil Das; Wonha Ryu; user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi all,
>
>
>
> The JSON version of the web UI is not officially supported; I don't
> believe this is documented anywhere.
>
>
>
> The alternative is to set `spark.eventLog.enabled` to true before running
> your application. This will create JSON SparkListenerEvents with details
> about each task and stage as a log file. Then you can easily reconstruct
> the web UI after the application has exited. This is what the standalone
> Master and the History Server does, actually. For local mode, you can use
> the latter to generate your UI after the fact. (This is documented here:
> http://spark.apache.org/docs/latest/monitoring.html).
>
>
>
> -Andrew
>
>
>
> 2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com>:
>
>
>
>      Hello,
>
>
>
>   Thanks for the link – this is for standalone, though, and most URLs
> don’t work for local.
>
>   I will look into deploying as standalone on a single node for testing
> and development.
>
>
>
>      Best, Oliver
>
>
>
> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com]
> *Sent:* Thursday, September 04, 2014 3:09 AM
> *To:* Ruebenacker, Oliver A
> *Cc:* Wonha Ryu; user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi
>
>
>
> You can see this doc
> <https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
> for all the available webUI ports.
>
>
>
> Yes there are ways to get the data metrics in Json format, One of them is
> below:
>
>
>
>  *​​*
>
> *http://webUI:8080/json/ <http://webUI:8080/json/>*  Or simply
>
> *​​*
>
> *curl webUI:8080/json/*
>
>
>
> There are some PRs about it you can read it over here
> https://github.com/apache/spark/pull/1682
>
>
>   Thanks
>
> Best Regards
>
>
>
> On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
>      Hello,
>
>
>
>   Interestingly, http://localhost:4040/metrics/json/ gives some numbers,
> but only a few which never seem to change during the application’s lifetime.
>
>
>
>   Either the web UI has some very strange limitations, or there are some
> URLs yet to be discovered that do something interesting.
>
>
>
>      Best,
>
>      Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha.ryu@gmail.com]
> *Sent:* Wednesday, September 03, 2014 4:27 PM
>
>
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hey Oliver,
>
>
>
> IIRC there's no JSON endpoint for application web UI. They only exist for
> cluster master and worker.
>
>
>
> - Wonha
>
>
>
>
>
> On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
>      Hello,
>
>
>
>   Thanks for the help! But I tried starting with “–master local[4]” and
> when I load http://localhost:4040/json I just get forwarded to
> http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
>
>
>
>      Best,
>
>      Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha.ryu@gmail.com]
> *Sent:* Wednesday, September 03, 2014 3:36 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi Oliver,
>
>
>
> Spark standalone master and worker support '/json' endpoint in web UI,
> which returns some of the information in JSON format.
>
> I wasn't able to find relevant documentation, though.
>
>
>
> - Wonha
>
>
>
> On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
>      Hello,
>
>
>
>   What is included in the Spark web UI? What are the available URLs? Can
> the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
>
>
>
>   Thanks!
>
>
>
>      Best,
>
>      Oliver
>
>
>
> Oliver Ruebenacker | Solutions Architect
>
>
>
> Altisource™
>
> 290 Congress St, 7th Floor | Boston, Massachusetts 02210
>
> P: (617) 728-5582 | ext: 275585
>
> Oliver.Ruebenacker@Altisource.com | www.Altisource.com
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>

Mime
View raw message