spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Date Sun, 17 Mar 2019 21:31:58 GMT
hi all,
i am having a bit trouble following the RCs this time. is RC6 still the one
being voted on? i see on github v2.4.1-rc7 and v2.4.1-rc8 but no email
threads.
thanks!
best, koert

On Mon, Mar 11, 2019 at 1:32 AM Xiao Li <lixiao@databricks.com> wrote:

> ++ Ryan Blue. What is your opinion regarding this Parquet issue?
>
> Thanks,
>
> Xiao
>
> On Sun, Mar 10, 2019 at 9:58 PM Wenchen Fan <cloud0fan@gmail.com> wrote:
>
>> Which version of Parquet has this bug? Maybe we can downgrade it.
>>
>> On Mon, Mar 11, 2019 at 10:34 AM Mark Hamstra <mark@clearstorydata.com>
>> wrote:
>>
>>> It worked in 2.3. We broke it with 2.4.0 and were informed of that
>>> regression late in the 2.4.0 release process. Since we didn't fix it before
>>> the 2.4.0 release, it should have been noted as a known issue. To now claim
>>> that there is no regression from 2.4.0 is a circular argument denying the
>>> existence of a known regression from 2.3.
>>>
>>> On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <srowen@gmail.com> wrote:
>>>
>>>> From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading
>>>> that:
>>>>
>>>> - this is a Parquet-Avro version conflict thing
>>>> - a downstream app wants different versions of Parquet and Avro than
>>>> Spark uses, which triggers it
>>>> - it doesn't work in 2.4.0
>>>>
>>>> It's not a regression from 2.4.0, which is the immediate question.
>>>> There isn't even a Parquet fix available.
>>>> But I'm not even seeing why this is excuse-making?
>>>>
>>>> On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <mark@clearstorydata.com>
>>>> wrote:
>>>> >
>>>> > Now wait... we created a regression in 2.4.0. Arguably, we should
>>>> have blocked that release until we had a fix; but the issue came up late
in
>>>> the release process and it looks to me like there wasn't an adequate fix
>>>> immediately available, so we did something bad and released 2.4.0 with a
>>>> known regression. Saying that there is now no regression from 2.4 is
>>>> tautological and no excuse for not taking in a fix -- and it looks like
>>>> that fix has been waiting for months.
>>>>
>>>
>
> --
> [image:
> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>

Mime
View raw message