spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Williams <ryan.blake.willi...@gmail.com>
Subject Spree: a live-updating web UI for Spark
Date Mon, 27 Jul 2015 22:14:31 GMT
Probably relevant to people on this list: on Friday I released a clone of
the Spark web UI built using Meteor <https://www.meteor.com/> so that
everything updates in real-time, saving you from endlessly refreshing the
page while jobs are running :) It can also serve as the UI for running as
well as completed applications, so you don't have to mess with the separate
history-server process if you don't want to.

*This blog post*
<http://www.hammerlab.org/2015/07/25/spree-58-a-live-updating-web-ui-for-spark/>
and
*the github repo* <https://github.com/hammerlab/spree> have lots of
information on how to use it.

It has two sub-components, JsonRelay
<https://github.com/hammerlab/spark-json-relay> and Slim
<https://github.com/hammerlab/slim>; the former sends SparkListenerEvent
<https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala#L33>s
out of Spark via a network socket, while the latter receives those events
and writes various stats about them to Mongo (like an external
JobProgressListener
<https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala>
that persists info to a database). You might find them to offer a better
way of storing information about running and completed Spark applications
than the event log files that Spark uses, and they can be used with or
without the real-time web UI.

Give them a try if they sound useful to you, and let me know if you have
questions or comments!

-Ryan

Mime
View raw message