Hi,

It might be worth including the fix blow in the next candidate?:
https://github.com/apache/spark/pull/24085


On Mon, Mar 18, 2019 at 4:54 PM DB Tsai <dbtsai@dbtsai.com.invalid> wrote:
Rc6 was failed due to the critical bug Xiao mentioned. Rc8 was cut to include those fixes, but I ran into some nexus infra issues when I tried to upload the jars into Apache repo. 
https://issues.apache.org/jira/plugins/servlet/mobile#issue/INFRA-18018 

The infra team reordered the key servers to prevent the timeout, and rebooted the server. I am running the build again to see if I can successfully upload Spark jars this time. Thanks!

On Sun, Mar 17, 2019 at 2:32 PM Koert Kuipers <koert@tresata.com> wrote:
hi all,
i am having a bit trouble following the RCs this time. is RC6 still the one being voted on? i see on github v2.4.1-rc7 and v2.4.1-rc8 but no email threads.
thanks!
best, koert

On Mon, Mar 11, 2019 at 1:32 AM Xiao Li <lixiao@databricks.com> wrote:
++ Ryan Blue. What is your opinion regarding this Parquet issue? 

Thanks,

Xiao

On Sun, Mar 10, 2019 at 9:58 PM Wenchen Fan <cloud0fan@gmail.com> wrote:
Which version of Parquet has this bug? Maybe we can downgrade it.

On Mon, Mar 11, 2019 at 10:34 AM Mark Hamstra <mark@clearstorydata.com> wrote:
It worked in 2.3. We broke it with 2.4.0 and were informed of that regression late in the 2.4.0 release process. Since we didn't fix it before the 2.4.0 release, it should have been noted as a known issue. To now claim that there is no regression from 2.4.0 is a circular argument denying the existence of a known regression from 2.3.  

On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <srowen@gmail.com> wrote:
From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading that:

- this is a Parquet-Avro version conflict thing
- a downstream app wants different versions of Parquet and Avro than
Spark uses, which triggers it
- it doesn't work in 2.4.0

It's not a regression from 2.4.0, which is the immediate question.
There isn't even a Parquet fix available.
But I'm not even seeing why this is excuse-making?

On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <mark@clearstorydata.com> wrote:
>
> Now wait... we created a regression in 2.4.0. Arguably, we should have blocked that release until we had a fix; but the issue came up late in the release process and it looks to me like there wasn't an adequate fix immediately available, so we did something bad and released 2.4.0 with a known regression. Saying that there is now no regression from 2.4 is tautological and no excuse for not taking in a fix -- and it looks like that fix has been waiting for months.


--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
--
- DB Sent from my iPhone


--
---
Takeshi Yamamuro