spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiao Li <lix...@databricks.com>
Subject Re: Getting the ball started on a 2.4.6 release
Date Thu, 23 Apr 2020 18:43:19 GMT
Hi, Holden,

We are trying to avoid backporting the improvement/cleanup PRs to the
maintenance releases, especially the core modules, like Spark Core and
SQL. For example, SPARK-26390 is a good example.

Xiao

On Thu, Apr 23, 2020 at 11:17 AM Holden Karau <holden@pigscanfly.ca> wrote:

> Tentatively I'm planning on this list to start backporting. If no one sees
> any issues with those I'll start to make backport JIRAs for them for
> tracking this afternoon.
> SPARK-26390       ColumnPruning rule should only do column pruning
> SPARK-25407       Allow nested access for non-existent field for Parquet
> file when nested pruning is enabled
> SPARK-25559       Remove the unsupported predicates in Parquet when
> possible
> SPARK-25860       Replace Literal(null, _) with FalseLiteral whenever
> possible
> SPARK-27514       Skip collapsing windows with empty window expressions
> SPARK-25338       Ensure to call super.beforeAll() and super.afterAll() in
> test cases
> SPARK-27138       Remove AdminUtils calls (fixes deprecation)
> SPARK-27981       Remove `Illegal reflective access` warning for
> `java.nio.Bits.unaligned()` in JDK9+
> SPARK-26095       Disable parallelization in make-distibution.sh. (Avoid
> build hanging)
> SPARK-25692       Remove static initialization of worker eventLoop
> handling chunk fetch requests within TransportContext. This fixes
> ChunkFetchIntegrationSuite as well
> SPARK-26306       More memory to de-flake SorterSuite
> SPARK-30199       Recover `spark.(ui|blockManager).port` from checkpoint
> SPARK-27676       InMemoryFileIndex should respect
> spark.sql.files.ignoreMissingFiles
> SPARK-31047       Improve file listing for ViewFileSystem
> SPARK-25595       Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES
> enabled
>
> Maybe:
> SPARK-27801       Delegate to ViewFileSystem during file listing correctly
>
> Not yet merged:
> SPARK-31485       Barrier execution hang if insufficient resources
>
> On Thu, Apr 23, 2020 at 9:13 AM Holden Karau <holden@pigscanfly.ca> wrote:
>
>>
>>
>> On Thu, Apr 23, 2020 at 9:07 AM edeesis <edeesis@gmail.com> wrote:
>>
>>> There's other information you can obtain from the Pod metadata on a
>>> describe
>>> than just from the logs, which are typically what's being printed by the
>>> Application itself.
>>
>> Would get pods -w -o yaml do the trick here or is there going to be
>> information that wouldn’t be captured that way?
>>
>>>
>>>
>>> I've also found that Spark has some trouble obtaining the reason for a
>>> K8S
>>> executor death (as evident by the
>>> spark.kubernetes.executor.lostCheck.maxAttempts config property)
>>>
>>> I admittedly don't know what should qualify for a backport, but
>>> considering
>>> 3.0 is a major upgrade (Scala version, et al), is there any room for for
>>> being more generous with backporting to 2.4?
>>
>> I’d like to revisit the conversation around a Spark 2.5 as a transitional
>> release. I know that some people are already effectively maintaining 2.4+
>> Selective new functionality backports internally. Maybe I’ll kick off that
>> discussion which we can have and that can help inform what we should be
>> putting in 2.4.
>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>


-- 
<https://databricks.com/sparkaisummit/north-america>

Mime
View raw message