Hi,

There's no better way to start a project than...github it :-) Create a new project, clone it and do dzieła! (= go ahead in Polish).

Jacek


On 27 Aug 2016 10:42 a.m., "Sivakumaran S" <siva.kumaran@icloud.com> wrote:

I would love to participate in developing a dashboard of some sort in lieu (or at least complement it)  of Spark UI .

Regards,

Sivakumaran S


On 27 Aug 2016 9:34 a.m., Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
Are we actually looking for a eal time dashboard of some sort for Spark UI interface?

After all one can think a real time dashboard can do this!

HTH



Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 26 August 2016 at 23:38, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
Thanks Jacek,

I will have a look. I think it is long overdue.

I mean we try to micro batch and stream everything below seconds but when it comes to help  monitor basics we are still miles behind :(

Cheers,

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 26 August 2016 at 23:21, Jacek Laskowski <jacek@japila.pl> wrote:
Hi Mich,

I don't think so. There is support for a UI page refresh but I haven't
seen it in use.

See StreamingPage [1] where it schedules refresh every 5 secs, i.e.
Some(5000). In SparkUIUtils.headerSparkPage [2] there is
refreshInterval but it's not used in any place in Spark.

Time to fill an JIRA issue?

What about REST API and httpie updating regularly [3]? Perhaps Metrics
with ConsoleSink [4]?

[1] https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L158
[2] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/UIUtils.scala#L202
[3] http://spark.apache.org/docs/latest/monitoring.html#rest-api
[4] http://spark.apache.org/docs/latest/monitoring.html#metrics

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Thu, Aug 25, 2016 at 11:55 AM, Mich Talebzadeh
<mich.talebzadeh@gmail.com> wrote:
> Hi,
>
> This may be already there.
>
> A spark job opens up a UI on port specified by --conf "spark.ui.port=${SP}"
> that defaults to 4040.
>
> However, on UI one needs to refresh the page to see the progress.
>
> Can this be polled so it is refreshed automatically
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed. The
> author will in no case be liable for any monetary damages arising from such
> loss, damage or destruction.
>
>