spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiao Li <lix...@databricks.com>
Subject Re: Getting the ball started on a 2.4.6 release
Date Thu, 23 Apr 2020 18:51:19 GMT
Actually, SPARK-26390 https://github.com/apache/spark/pull/23343 is just a
small clean up. I do not think it fixes any correctness bugs.

I think we should discuss your backport plans one by one with the PR
authors and reviewers, since most of them are not closely following the dev
list.

Xiao


On Thu, Apr 23, 2020 at 11:46 AM Holden Karau <holden@pigscanfly.ca> wrote:

> I included 26390 as a candidate since it sounded like it bordered on a
> correctness/expected behaviour fix (eg columpruning rule doing more than
> column pruning), but if it’s too big a change happy to drop that one.
>
> On Thu, Apr 23, 2020 at 11:43 AM Xiao Li <lixiao@databricks.com> wrote:
>
>> Hi, Holden,
>>
>> We are trying to avoid backporting the improvement/cleanup PRs to the
>> maintenance releases, especially the core modules, like Spark Core and
>> SQL. For example, SPARK-26390 is a good example.
>>
>> Xiao
>>
>> On Thu, Apr 23, 2020 at 11:17 AM Holden Karau <holden@pigscanfly.ca>
>> wrote:
>>
>>> Tentatively I'm planning on this list to start backporting. If no one
>>> sees any issues with those I'll start to make backport JIRAs for them for
>>> tracking this afternoon.
>>> SPARK-26390       ColumnPruning rule should only do column pruning
>>> SPARK-25407       Allow nested access for non-existent field for Parquet
>>> file when nested pruning is enabled
>>> SPARK-25559       Remove the unsupported predicates in Parquet when
>>> possible
>>> SPARK-25860       Replace Literal(null, _) with FalseLiteral whenever
>>> possible
>>> SPARK-27514       Skip collapsing windows with empty window expressions
>>> SPARK-25338       Ensure to call super.beforeAll() and super.afterAll()
>>> in test cases
>>> SPARK-27138       Remove AdminUtils calls (fixes deprecation)
>>> SPARK-27981       Remove `Illegal reflective access` warning for
>>> `java.nio.Bits.unaligned()` in JDK9+
>>> SPARK-26095       Disable parallelization in make-distibution.sh. (Avoid
>>> build hanging)
>>> SPARK-25692       Remove static initialization of worker eventLoop
>>> handling chunk fetch requests within TransportContext. This fixes
>>> ChunkFetchIntegrationSuite as well
>>> SPARK-26306       More memory to de-flake SorterSuite
>>> SPARK-30199       Recover `spark.(ui|blockManager).port` from checkpoint
>>> SPARK-27676       InMemoryFileIndex should respect
>>> spark.sql.files.ignoreMissingFiles
>>> SPARK-31047       Improve file listing for ViewFileSystem
>>> SPARK-25595       Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES
>>> enabled
>>>
>>> Maybe:
>>> SPARK-27801       Delegate to ViewFileSystem during file listing
>>> correctly
>>>
>>> Not yet merged:
>>> SPARK-31485       Barrier execution hang if insufficient resources
>>>
>>> On Thu, Apr 23, 2020 at 9:13 AM Holden Karau <holden@pigscanfly.ca>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Apr 23, 2020 at 9:07 AM edeesis <edeesis@gmail.com> wrote:
>>>>
>>>>> There's other information you can obtain from the Pod metadata on a
>>>>> describe
>>>>> than just from the logs, which are typically what's being printed by
>>>>> the
>>>>> Application itself.
>>>>
>>>> Would get pods -w -o yaml do the trick here or is there going to be
>>>> information that wouldn’t be captured that way?
>>>>
>>>>>
>>>>>
>>>>> I've also found that Spark has some trouble obtaining the reason for
a
>>>>> K8S
>>>>> executor death (as evident by the
>>>>> spark.kubernetes.executor.lostCheck.maxAttempts config property)
>>>>>
>>>>> I admittedly don't know what should qualify for a backport, but
>>>>> considering
>>>>> 3.0 is a major upgrade (Scala version, et al), is there any room for
>>>>> for
>>>>> being more generous with backporting to 2.4?
>>>>
>>>> I’d like to revisit the conversation around a Spark 2.5 as a
>>>> transitional release. I know that some people are already effectively
>>>> maintaining 2.4+ Selective new functionality backports internally. Maybe
>>>> I’ll kick off that discussion which we can have and that can help inform
>>>> what we should be putting in 2.4.
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>
>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>>
>> --
>> <https://databricks.com/sparkaisummit/north-america>
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>


-- 
<https://databricks.com/sparkaisummit/north-america>

Mime
View raw message