spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Is there anyway Spark UI is set to poll and refreshes itself
Date Fri, 26 Aug 2016 22:38:02 GMT
Thanks Jacek,

I will have a look. I think it is long overdue.

I mean we try to micro batch and stream everything below seconds but when
it comes to help  monitor basics we are still miles behind :(

Cheers,

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 26 August 2016 at 23:21, Jacek Laskowski <jacek@japila.pl> wrote:

> Hi Mich,
>
> I don't think so. There is support for a UI page refresh but I haven't
> seen it in use.
>
> See StreamingPage [1] where it schedules refresh every 5 secs, i.e.
> Some(5000). In SparkUIUtils.headerSparkPage [2] there is
> refreshInterval but it's not used in any place in Spark.
>
> Time to fill an JIRA issue?
>
> What about REST API and httpie updating regularly [3]? Perhaps Metrics
> with ConsoleSink [4]?
>
> [1] https://github.com/apache/spark/blob/master/streaming/
> src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L158
> [2] https://github.com/apache/spark/blob/master/core/src/
> main/scala/org/apache/spark/ui/UIUtils.scala#L202
> [3] http://spark.apache.org/docs/latest/monitoring.html#rest-api
> [4] http://spark.apache.org/docs/latest/monitoring.html#metrics
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Thu, Aug 25, 2016 at 11:55 AM, Mich Talebzadeh
> <mich.talebzadeh@gmail.com> wrote:
> > Hi,
> >
> > This may be already there.
> >
> > A spark job opens up a UI on port specified by --conf
> "spark.ui.port=${SP}"
> > that defaults to 4040.
> >
> > However, on UI one needs to refresh the page to see the progress.
> >
> > Can this be polled so it is refreshed automatically
> >
> > Thanks
> >
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn
> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw
> >
> >
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > Disclaimer: Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> The
> > author will in no case be liable for any monetary damages arising from
> such
> > loss, damage or destruction.
> >
> >
>

Mime
View raw message