spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Sadikov <ivan.sadi...@gmail.com>
Subject Spark history server running on Mongo
Date Tue, 18 Jul 2017 07:56:52 GMT
Hello everyone!

I have been working on Spark history server that uses MongoDB as a
datastore for processed events to iterate on idea that Spree project uses
for Spark UI. Project was originally designed to improve on standalone
history server with reduced memory footprint.

Project lives here: https://github.com/lightcopy/history-server

These are just very early days of the project, sort of pre-alpha (some
features are missing, and metrics in some failed jobs cases are
questionable). Code is being tested on several 8gb and 2gb logs and aims to
lower resource usage since we run history server together with several
other systems.

Would greatly appreciate any feedback on repository (issues/pull
requests/suggestions/etc.). Thanks a lot!


Cheers,

Ivan

Mime
View raw message