spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: Spark history server running on Mongo
Date Tue, 18 Jul 2017 16:18:40 GMT
See SPARK-18085. That has much of the same goals re: SHS resource
usage, and also provides a (currently non-public) API where you could
just create a MongoDB implementation if you want.

On Tue, Jul 18, 2017 at 12:56 AM, Ivan Sadikov <ivan.sadikov@gmail.com> wrote:
> Hello everyone!
>
> I have been working on Spark history server that uses MongoDB as a datastore
> for processed events to iterate on idea that Spree project uses for Spark
> UI. Project was originally designed to improve on standalone history server
> with reduced memory footprint.
>
> Project lives here: https://github.com/lightcopy/history-server
>
> These are just very early days of the project, sort of pre-alpha (some
> features are missing, and metrics in some failed jobs cases are
> questionable). Code is being tested on several 8gb and 2gb logs and aims to
> lower resource usage since we run history server together with several other
> systems.
>
> Would greatly appreciate any feedback on repository (issues/pull
> requests/suggestions/etc.). Thanks a lot!
>
>
> Cheers,
>
> Ivan
>



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message