spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Log analyzer and other Spark tools
Date Tue, 18 Mar 2014 00:34:06 GMT
Take a look at the SparkListener API included in Spark, you can use it to capture various events.
There’s also this pull request: https://github.com/apache/spark/pull/42 that will persist
application logs and let you rebuild the web UI after the app runs. It uses the same API to
log events.

Matei

On Mar 17, 2014, at 7:35 AM, Roman Pastukhov <metaignatich@gmail.com> wrote:

> Hi.
> 
> We're thinking about writing a tool that would read Spark logs and output cache contents
at some point in time (e.g. if you want to see what data fills the cache and whether some
of it may be unpersisted to improve performance).
> 
> Are there similar projects that already exist? Is there a list of Spark-related tools?
There is Spark debugger/SRD (https://github.com/mesos/spark/wiki/Spark-Debugger, http://spark-replay-debugger-overview.readthedocs.org/en/latest/)
but I couldn't find any links to them on the Spark project site.


Mime
View raw message