spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruebenacker, Oliver A" <Oliver.Ruebenac...@altisource.com>
Subject RE: Web UI
Date Fri, 05 Sep 2014 13:50:06 GMT

     Hello,

  Thanks for the explanation. So events are stored internally as JSON, but there is no official
support for having Spark serve that JSON via HTTP? So if I wanted to write an app that monitors
Spark, I would either have to scrape the web UI in HTML or rely on unofficial JSON features?
That is quite surprising, because I would expect dumping out the JSON would be easier for
Spark developers to implement than converting it to HTML.

  Do I get that right? Should I make a feature request? Thanks!

     Best, Oliver

From: Andrew Or [mailto:andrew@databricks.com]
Sent: Thursday, September 04, 2014 2:11 PM
To: Ruebenacker, Oliver A
Cc: Akhil Das; Wonha Ryu; user@spark.apache.org
Subject: Re: Web UI

Hi all,

The JSON version of the web UI is not officially supported; I don't believe this is documented
anywhere.

The alternative is to set `spark.eventLog.enabled` to true before running your application.
This will create JSON SparkListenerEvents with details about each task and stage as a log
file. Then you can easily reconstruct the web UI after the application has exited. This is
what the standalone Master and the History Server does, actually. For local mode, you can
use the latter to generate your UI after the fact. (This is documented here: http://spark.apache.org/docs/latest/monitoring.html).

-Andrew

2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A <Oliver.Ruebenacker@altisource.com<mailto:Oliver.Ruebenacker@altisource.com>>:

     Hello,

  Thanks for the link – this is for standalone, though, and most URLs don’t work for local.
  I will look into deploying as standalone on a single node for testing and development.

     Best, Oliver

From: Akhil Das [mailto:akhil@sigmoidanalytics.com<mailto:akhil@sigmoidanalytics.com>]
Sent: Thursday, September 04, 2014 3:09 AM
To: Ruebenacker, Oliver A
Cc: Wonha Ryu; user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Web UI

Hi

You can see this doc<https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
for all the available webUI ports.

Yes there are ways to get the data metrics in Json format, One of them is below:

​​
http://webUI:8080/json/  Or simply
​​
curl webUI:8080/json/

There are some PRs about it you can read it over here https://github.com/apache/spark/pull/1682

Thanks
Best Regards

On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A <Oliver.Ruebenacker@altisource.com<mailto:Oliver.Ruebenacker@altisource.com>>
wrote:

     Hello,

  Interestingly, http://localhost:4040/metrics/json/ gives some numbers, but only a few which
never seem to change during the application’s lifetime.

  Either the web UI has some very strange limitations, or there are some URLs yet to be discovered
that do something interesting.

     Best,
     Oliver


From: Wonha Ryu [mailto:wonha.ryu@gmail.com<mailto:wonha.ryu@gmail.com>]
Sent: Wednesday, September 03, 2014 4:27 PM

To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Web UI

Hey Oliver,

IIRC there's no JSON endpoint for application web UI. They only exist for cluster master and
worker.

- Wonha


On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <Oliver.Ruebenacker@altisource.com<mailto:Oliver.Ruebenacker@altisource.com>>
wrote:

     Hello,

  Thanks for the help! But I tried starting with “–master local[4]” and when I load
http://localhost:4040/json I just get forwarded to http://localhost:4040/stages/, and it’s
all human-readable HTML, no JSON.

     Best,
     Oliver


From: Wonha Ryu [mailto:wonha.ryu@gmail.com<mailto:wonha.ryu@gmail.com>]
Sent: Wednesday, September 03, 2014 3:36 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Web UI

Hi Oliver,

Spark standalone master and worker support '/json' endpoint in web UI, which returns some
of the information in JSON format.
I wasn't able to find relevant documentation, though.

- Wonha

On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <Oliver.Ruebenacker@altisource.com<mailto:Oliver.Ruebenacker@altisource.com>>
wrote:

     Hello,

  What is included in the Spark web UI? What are the available URLs? Can the information be
obtained in a machine-readable way (e.g. JSON, XML, etc)?

  Thanks!

     Best,
     Oliver

Oliver Ruebenacker | Solutions Architect

Altisource™
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582<tel:%28617%29%20728-5582> | ext: 275585
Oliver.Ruebenacker@Altisource.com<mailto:Oliver.Ruebenacker@Altisource.com> | www.Altisource.com<http://www.Altisource.com>

***********************************************************************************************************************

This email message and any attachments are intended solely for the use of the addressee. If
you are not the intended recipient, you are prohibited from reading, disclosing, reproducing,
distributing, disseminating or otherwise using this transmission. If you have received this
message in error, please promptly notify the sender by reply email and immediately delete
this message from your system.
This message and any attachments may contain information that is confidential, privileged
or exempt from disclosure. Delivery of this message to any person other than the intended
recipient is not intended to waive any right or privilege. Message transmission is not guaranteed
to be secure or free of software viruses.
***********************************************************************************************************************

***********************************************************************************************************************

This email message and any attachments are intended solely for the use of the addressee. If
you are not the intended recipient, you are prohibited from reading, disclosing, reproducing,
distributing, disseminating or otherwise using this transmission. If you have received this
message in error, please promptly notify the sender by reply email and immediately delete
this message from your system.
This message and any attachments may contain information that is confidential, privileged
or exempt from disclosure. Delivery of this message to any person other than the intended
recipient is not intended to waive any right or privilege. Message transmission is not guaranteed
to be secure or free of software viruses.
***********************************************************************************************************************

***********************************************************************************************************************

This email message and any attachments are intended solely for the use of the addressee. If
you are not the intended recipient, you are prohibited from reading, disclosing, reproducing,
distributing, disseminating or otherwise using this transmission. If you have received this
message in error, please promptly notify the sender by reply email and immediately delete
this message from your system.
This message and any attachments may contain information that is confidential, privileged
or exempt from disclosure. Delivery of this message to any person other than the intended
recipient is not intended to waive any right or privilege. Message transmission is not guaranteed
to be secure or free of software viruses.
***********************************************************************************************************************

***********************************************************************************************************************

This email message and any attachments are intended solely for the use of the addressee. If
you are not the intended recipient, you are prohibited from reading, disclosing, reproducing,
distributing, disseminating or otherwise using this transmission. If you have received this
message in error, please promptly notify the sender by reply email and immediately delete
this message from your system.
This message and any attachments may contain information that is confidential, privileged
or exempt from disclosure. Delivery of this message to any person other than the intended
recipient is not intended to waive any right or privilege. Message transmission is not guaranteed
to be secure or free of software viruses.
***********************************************************************************************************************

***********************************************************************************************************************

This email message and any attachments are intended solely for the use of the addressee. If
you are not the intended recipient, you are prohibited from reading, disclosing, reproducing,
distributing, disseminating or otherwise using this transmission. If you have received this
message in error, please promptly notify the sender by reply email and immediately delete
this message from your system. This message and any attachments may contain information that
is confidential, privileged or exempt from disclosure. Delivery of this message to any person
other than the intended recipient is not intended to waive any right or privilege. Message
transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
Mime
View raw message