spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dongjoon Hyun <dongjoon.h...@gmail.com>
Subject Re: [VOTE] Release Apache Spark 2.4.4 (RC1)
Date Sat, 24 Aug 2019 14:41:45 GMT
FYI, three more patches landed to `branch-2.4` until yesterday.

[SPARK-28642][SQL][2.4] Hide credentials in show create table
[SPARK-27330][SS][2.4] support task abort in foreach writer
[SPARK-28025][SS][2.4] Fix FileContextBasedCheckpointFileManager leaking
crc files


For some non-blocker issues like `[SPARK-28778][MESOS] Fixed executors
advertised address ...`,
we can have it if it lands before `2.4.4-rc2` tag creation. I'll make
`2.4.4-rc2` tag tomorrow.

Please let me know if there is blocker issues.

Bests,
Dongjoon.


On Thu, Aug 22, 2019 at 9:28 AM Dongjoon Hyun <dongjoon.hyun@gmail.com>
wrote:

> Hi, All.
>
> This 2.4.4 RC1 vote fails and the reported PRs are merged to `branch-2.4`.
> The following is the commit list since `v2.4.4-rc1` tag.
>
> Preparing development version 2.4.5-SNAPSHOT
> [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8
> [SPARK-28775][CORE][TESTS] Skip date 8633 in Kwajalein due to changes in
> tzdata2018i that only some JDK 8s use
> [SPARK-28777][PYTHON][DOCS] Fix format_string doc string with the correct
> parameters
> [SPARK-28650][SS][DOC] Correct explanation of guarantee for ForeachWriter
> [SPARK-26895][CORE][2.4] prepareSubmitEnvironment should be called within
> doAs for proxy users
> [SPARK-28699][SQL] Disable using radix sort for ShuffleExchangeExec in
> repartition case
> [SPARK-28844][SQL] Fix typo in SQLConf FILE_COMRESSION_FACTOR
> [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
> [SPARK-28699][CORE][2.4] Fix a corner case for aborting indeterminate stage
>
>
> Please let me know if we need more patches.
> I'm going to cut `2.4.4-rc2` tag during weekend and starts RC2 on next
> Monday.
>
> Bests,
> Dongjoon.
>
>
> On Tue, Aug 20, 2019 at 5:01 AM Sean Owen <srowen@apache.org> wrote:
>
>> Sounds fine, we probably needed SPARK-28775 anyway. I merged that and
>> SPARK-28749. It looks like it's just the one you're talking about
>> right now, SPARK-28699.
>> The rest of the tests seemed to pass OK, release looks good, but bears
>> more testing by everyone out there before a next RC.
>>
>> On Tue, Aug 20, 2019 at 12:10 AM Wenchen Fan <cloud0fan@gmail.com> wrote:
>> >
>> > Unfortunately, I need to -1.
>> >
>> > Recently we found that the repartition correctness bug can still be
>> reproduced. The root cause has been identified and there are 2 PRs to fix 2
>> related issues:
>> > https://github.com/apache/spark/pull/25491
>> > https://github.com/apache/spark/pull/25498
>> >
>> > I think we should have this fix in 2.3 and 2.4.
>> >
>> > Thanks,
>> > Wenchen
>> >
>> > On Tue, Aug 20, 2019 at 7:32 AM Dongjoon Hyun <dongjoon.hyun@gmail.com>
>> wrote:
>> >>
>> >> Thank you for testing, Sean and Herman.
>> >>
>> >> There are three reporting until now.
>> >>
>> >> 1. SPARK-28775 is for JDK 8u221+ testing at Apache Spark 3.0/2.4/2.3.
>> >> 2. SPARK-28749 is for Scala 2.12 Python testing at Apache Spark 2.4
>> only.
>> >> 3. SPARK-28699 is for disabling radix sort for ShuffleExchangeExec at
>> Apache Spark 3.0/2.4/2.3.
>> >>
>> >> Both (1) and (2) are nice-to-have and test-only fixes. (3) could be a
>> correctness issue, but it seems that there are some other approaches.
>> >> I'm monitoring all reports. Let's see. For now, I'd like to continue
>> 2.4.4 RC1 voting for more testing.
>> >>
>> >> Bests,
>> >> Dongjoon.
>>
>

Mime
View raw message