Hi,

BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll see 3.1.1-rc1?

[1] https://github.com/apache/spark/tree/v3.1.0-rc1



On Thu, Jan 7, 2021 at 3:12 PM Jacek Laskowski <jacek@japila.pl> wrote:
Hi,

I'm just reading this now. I'm for 3.1.1 with no 3.1.0 but the news that we're skipping that particular release. Gonna be more fun! :)



On Thu, Jan 7, 2021 at 6:13 AM Hyukjin Kwon <gurwls223@gmail.com> wrote:
Thank you Holden and Wenchen!

Let me:
- prepare a PR for news in spark-website first about 3.1.0 accident late tonight (in KST)
- and start to prepare 3.1.1 probably in few more days like next monday in case other people have different thoughts



2021년 1월 7일 (목) 오후 2:04, Holden Karau <holden@pigscanfly.ca>님이 작성:
I think that posting the 3.1.0 maven release was an accident and we're going to 3.1.1 RCs is the right step forward.
I'd ask for maybe a day before cutting the 3.1.1 release, I think https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at first I thought it was just a test issue, but Dongjoon pointed out the NPE happens in prod too).

I'd also like to echo the: it's totally ok we all make mistakes especially in partially manual & partially automated environments, I've created a bunch of RCs labels without recognizing they were getting pushed automatically.

On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan <cloud0fan@gmail.com> wrote:
I agree with Jungtaek that people are likely to be biased when testing 3.1.0. At least this will not be the same community-blessed release as previous ones, because the voting is already affected by the fact that 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.

On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon <gurwls223@gmail.com> wrote:
Okay, let me just start to prepare 3.1.1. I think that will address all concerns except that 3.1.0 will remain in Maven as incomplete.
By right, removal in the Maven repo is disallowed. Overwrite is possible as far as I know but other mirrors that maintain cache will get affected.
Maven is one of the downstream publish channels, and we haven't officially announced and published it to Apache repo anyway.
I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away.
Are we all good with this?



2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon <gurwls223@gmail.com>님이 작성:
I think that It would be great though if we have a clear blocker that makes the release pointless if we want to drop this RC practically given that we will schedule 3.1.1 faster - non-regression bug fixes will be delivered to end users relatively fast.
That would make it clear which option we should take. I personally don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's response anyway.


2021년 1월 7일 (목) 오후 1:03, Sean Owen <srowen@gmail.com>님이 작성:
I don't agree the first two are blockers for reasons I gave earlier.
Those two do look like important issues - are they regressions from 3.0.1?
I do agree we'd probably cut a new RC for those in any event, so agree with the plan to drop 3.1.0 (if the Maven release can't be overwritten)

On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun <dongjoon.hyun@gmail.com> wrote:
Before we discover the pre-uploaded artifacts, both Jungtaek and Hyukjin already made two blockers shared here.
IIUC, it meant implicitly RC1 failure at that time.

In addition to that, there are two correctness issues. So, I made up my mind to cast -1 for this RC1 before joining this thread.

SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache (committed after tagging)
SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache (PR is under review)

Although the above issues are not regression, those are enough for me to give -1 for 3.1.0 RC1.

On Wed, Jan 6, 2021 at 3:52 PM Sean Owen <srowen@gmail.com> wrote:
I just don't see a reason to believe there's a rush? just test it as normal? I did, you can too, etc.
Or specifically what blocks the current RC?


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9