spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Getting the ball started on a 2.4.6 release
Date Thu, 23 Apr 2020 16:13:25 GMT
On Thu, Apr 23, 2020 at 9:07 AM edeesis <edeesis@gmail.com> wrote:

> There's other information you can obtain from the Pod metadata on a
> describe
> than just from the logs, which are typically what's being printed by the
> Application itself.

Would get pods -w -o yaml do the trick here or is there going to be
information that wouldn’t be captured that way?

>
>
> I've also found that Spark has some trouble obtaining the reason for a K8S
> executor death (as evident by the
> spark.kubernetes.executor.lostCheck.maxAttempts config property)
>
> I admittedly don't know what should qualify for a backport, but considering
> 3.0 is a major upgrade (Scala version, et al), is there any room for for
> being more generous with backporting to 2.4?

I’d like to revisit the conversation around a Spark 2.5 as a transitional
release. I know that some people are already effectively maintaining 2.4+
Selective new functionality backports internally. Maybe I’ll kick off that
discussion which we can have and that can help inform what we should be
putting in 2.4.

>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Mime
View raw message