spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Sadikov <ivan.sadi...@gmail.com>
Subject Re: Spark history server running on Mongo
Date Wed, 19 Jul 2017 02:21:57 GMT
Hi Marcelo,

Thanks for the reference, again. I looked at your code - really great work!
I had to replace Spark distribution to use it though - could not figure out
how to build it separately.

Repository that I linked to does not require rebuilding Spark and could be
used with current distribution, which is preferable in my case.


Kind regards,

Ivan



On Wed, 19 Jul 2017 at 4:44 AM, Ivan Sadikov <ivan.sadikov@gmail.com> wrote:

> Thanks for JIRA ticket reference! Frankly, I was aware of this work, but
> didn't know that there was an API for storage implementation.
>
> Will try exploring that as well, thanks!
> On Wed, 19 Jul 2017 at 4:18 AM, Marcelo Vanzin <vanzin@cloudera.com>
> wrote:
>
>> See SPARK-18085. That has much of the same goals re: SHS resource
>> usage, and also provides a (currently non-public) API where you could
>> just create a MongoDB implementation if you want.
>>
>> On Tue, Jul 18, 2017 at 12:56 AM, Ivan Sadikov <ivan.sadikov@gmail.com>
>> wrote:
>> > Hello everyone!
>> >
>> > I have been working on Spark history server that uses MongoDB as a
>> datastore
>> > for processed events to iterate on idea that Spree project uses for
>> Spark
>> > UI. Project was originally designed to improve on standalone history
>> server
>> > with reduced memory footprint.
>> >
>> > Project lives here: https://github.com/lightcopy/history-server
>> >
>> > These are just very early days of the project, sort of pre-alpha (some
>> > features are missing, and metrics in some failed jobs cases are
>> > questionable). Code is being tested on several 8gb and 2gb logs and
>> aims to
>> > lower resource usage since we run history server together with several
>> other
>> > systems.
>> >
>> > Would greatly appreciate any feedback on repository (issues/pull
>> > requests/suggestions/etc.). Thanks a lot!
>> >
>> >
>> > Cheers,
>> >
>> > Ivan
>> >
>>
>>
>>
>> --
>> Marcelo
>>
>

Mime
View raw message