spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability
Date Tue, 01 Aug 2017 09:18:58 GMT
(Direct link to design doc, linked from JIRA)

I know Marcelo has looked closely at this issue for a long while and trust
his judgment about what needs to be fixed, and how. I know he has a good
view into practical pain points like this from customer deployments.

I read the whole doc. Although I have not followed the SHS issues closely,
the recounting of issues makes sense to me, as well as the stated goals.

There is a considerable amount of change proposed here, but it's broken
down into milestones. The change doesn't seem excessive given what needs to
be improved -- it does need a rearchitecting.

Now, some minor comments

+1 to reusing a 'database' library already used by Spark.
Is 'spark-ui' too broad? doesn't sound like this module would actually
house all the UIs. spark-shs-ui or something?
Good that this can be implemented in parallel to the existing mechanism for
the initial milestones.
So the "Mx" milestones are essentially required, in your view? and the
"SMx" are optional and stand-alone?

So yes I have no concerns, trust the analysis, think it's a real problem
that will take a sustained effort. +1

On Mon, Jul 31, 2017 at 6:28 PM Marcelo Vanzin <> wrote:

> Hey all,
> Following the SPIP process, I'm putting this SPIP up for a vote. It's
> been open for comments as an SPIP for about 3 weeks now, and had been
> open without the SPIP label for about 9 months before that. There has
> been no new feedback since it was tagged as an SPIP, so I'm assuming
> all the people who looked at it are OK with the current proposal.
> The vote will be up for the next 72 hours. Please reply with your vote:
> +1: Yeah, let's go forward and implement the SPIP.
> +0: Don't really care.
> -1: I don't think this is a good idea because of the following
> technical reasons.
> Thanks!
> --
> Marcelo
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message