spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dongjoon Hyun <dongjoon.h...@gmail.com>
Subject Re: Spark 2.4.5 RC2 Preparation Status
Date Wed, 29 Jan 2020 22:09:32 GMT
Got it. Thanks!

Bests,
Dongjoon.

On Wed, Jan 29, 2020 at 1:40 PM Sean Owen <srowen@gmail.com> wrote:

> OK, we can wait a tick to confirm there aren't strong objections.
> I suppose I'd prefer someone who knows
> https://issues.apache.org/jira/browse/SPARK-28344 to confirm it was
> either erroneously targeted to 2.4, or else it's valid, but, not
> critical for the RC. Hearing nothing else shortly, I'd untarget it.
>
> SPARK-29578 is a tiny low-risk test change but probably worth picking
> up to avoid failing on certain JDKs during testing. I'll make a
> back-port, as this should be noncontroversial. (Not sure why I didn't
> backport originally)
>
> On Wed, Jan 29, 2020 at 3:27 PM Dongjoon Hyun <dongjoon.hyun@gmail.com>
> wrote:
> >
> > Thanks, Sean.
> >
> > If there is no further objection to the mailing list,
> > could you remove the `Target Version: 2.4.5` from the followings?
> >
> >     SPARK-28344 Fail the query if detect ambiguous self join
> >     SPARK-29578 JDK 1.8.0_232 timezone updates cause "Kwajalein" test
> failures again
> >
> > Then, after the regular RC preparation testing including the manual
> integration tests,
> > I can roll 2.4.5 RC2 next Monday (Feb. 3rd, PST) and all late blocker
> patches will block 2.4.6 instead of causing RC failure.
> >
> > Bests,
> > Dongjoon.
> >
> >
> > On Wed, Jan 29, 2020 at 12:16 PM Sean Owen <srowen@gmail.com> wrote:
> >>
> >> OK, that's specific. It's always a judgment call whether to hold the
> release train for one more fix or not. Depends on how impactful it is (harm
> of releasing without it), and how big it is (harm of delaying release of
> other fixes further). I think we tend to weight regressions from a previous
> 2.4.x release more heavily; those are typically Blockers, otherwise not.
> Otherwise once RCs start, we're driving primarily to a no-Blocker release.
> The default should be to punt to 2.4.6 -- which can come relatively soon if
> one wants.
> >>
> >> SPARK-28125 is not even a bug, I'd argue, let alone Blocker. Looks like
> it was marked 'correctness' by the reporter. It's always been the case
> since Spark 1.0 (i.e. not a regression) that RDDs need to be deterministic
> for most of the semantics one expects to work out. If it isn't, many bets
> are off. I get that this is a 'gotcha', but it isn't even about the
> randomSplit. If anything recomputes the RDD, it could be different.
> >>
> >> SPARK-28067, I don't know anything about, but also is being reported as
> not a 2.4.x regression, and I don't see anyone working on it. For that
> reason, not sure it's a Blocker for 2.4.x.
> >>
> >> SPARK-30310 is not a 2.4.x regression either, nor particularly critical
> IMHO. Doesn't mean we can't back-port it to 2.4 though, and it's 'done' (in
> master)
> >>
> >> Anything else? not according to JIRA at least.
> >>
> >> I think it's valid to continue with RC2 assuming none of these are
> necessary for 2.4.5.
> >> It's not wrong to 'wait' if there are strong feelings about something,
> but, if we can't see a reason to expect the situation changes in a week, 2
> weeks, then, why? The release of 2.4.5 nowish doesn't necessarily make the
> release of said fix much further away -- in 2.4.6.
> >>
> >> On Wed, Jan 29, 2020 at 1:28 PM Dongjoon Hyun <dongjoon.hyun@gmail.com>
> wrote:
> >>>
> >>>     > SPARK-28125 dataframes created by randomSplit have overlapping
> rows
> >>>     >     Seems like something we should fix
> >>>     > SPARK-28067 Incorrect results in decimal aggregation with
> whole-stage code gen enabled
> >>>     >     Seems like we should fix
> >>>
> >>> Here, I'm trying to narrow down our focus to the issues with `Explicit
> Target Version` and continue to release. In other words, as a release
> manager, I hope I can officially ignore the other correctness issues which
> is not targeting to 2.4.5 explicitly.
> >>>
> >>> Most correctness issues are long-standing and cause behavior changes.
> During maintenance RC vote, for those kind of issues, I hope we set the
> Target Version `2.4.6` instead of casting a veto RC. It's the same policy
> with Fix Version. During RC vote period, Fix Version is set to the next
> version `2.4.6` instead of the current RC `2.4.5`. Since maintenance
> happens more frequently, I believe that's okay.
> >>>>
> >>>>
>

Mime
View raw message