spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Is there anyway Spark UI is set to poll and refreshes itself
Date Fri, 26 Aug 2016 22:21:44 GMT
Hi Mich,

I don't think so. There is support for a UI page refresh but I haven't
seen it in use.

See StreamingPage [1] where it schedules refresh every 5 secs, i.e.
Some(5000). In SparkUIUtils.headerSparkPage [2] there is
refreshInterval but it's not used in any place in Spark.

Time to fill an JIRA issue?

What about REST API and httpie updating regularly [3]? Perhaps Metrics
with ConsoleSink [4]?

[1] https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L158
[2] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/UIUtils.scala#L202
[3] http://spark.apache.org/docs/latest/monitoring.html#rest-api
[4] http://spark.apache.org/docs/latest/monitoring.html#metrics

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Thu, Aug 25, 2016 at 11:55 AM, Mich Talebzadeh
<mich.talebzadeh@gmail.com> wrote:
> Hi,
>
> This may be already there.
>
> A spark job opens up a UI on port specified by --conf "spark.ui.port=${SP}"
> that defaults to 4040.
>
> However, on UI one needs to refresh the page to see the progress.
>
> Can this be polled so it is refreshed automatically
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed. The
> author will in no case be liable for any monetary damages arising from such
> loss, damage or destruction.
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message