spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Blue <>
Subject Re: Spark 2.4.2
Date Tue, 16 Apr 2019 23:55:02 GMT
Spark has a lot of strange behaviors already that we don't fix in patch
releases. And bugs aren't usually fixed with a configuration flag to turn
on the fix.

That said, I don't have a problem with this commit making it into a patch
release. This is a small change and looks safe enough to me. I was just a
little surprised since I was expecting a correctness issue if this is
prompting a release. I'm definitely on the side of case-by-case judgments
on what to allow in patch releases and this looks fine.

On Tue, Apr 16, 2019 at 4:27 PM Michael Armbrust <>

> I would argue that its confusing enough to a user for options from
> DataFrameWriter to be silently dropped when instantiating the data source
> to consider this a bug.  They asked for partitioning to occur, and we are
> doing nothing (not even telling them we can't).  I was certainly surprised
> by this behavior.  Do you have a different proposal about how this should
> be handled?
> On Tue, Apr 16, 2019 at 4:23 PM Ryan Blue <> wrote:
>> Is this a bug fix? It looks like a new feature to me.
>> On Tue, Apr 16, 2019 at 4:13 PM Michael Armbrust <>
>> wrote:
>>> Hello All,
>>> I know we just released Spark 2.4.1, but in light of fixing SPARK-27453
>>> <> I was wondering if
>>> it might make sense to follow up quickly with 2.4.2.  Without this fix its
>>> very hard to build a datasource that correctly handles partitioning without
>>> using unstable APIs.  There are also a few other fixes that have trickled
>>> in since 2.4.1.
>>> If there are no objections, I'd like to start the process shortly.
>>> Michael
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix

Ryan Blue
Software Engineer

View raw message