spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <sro...@gmail.com>
Subject Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all active branches
Date Wed, 10 Apr 2019 22:57:30 GMT
My guess is that branch 2.3, 2.4 still work with Python 3.6 even if
we're not testing that. If we're only testing against 3.6, I guess we
wouldn't detect if something broke for 3.4 which is still 'supported'
in 2.3.x, 2.4.x? For 3.x, no problem, let's raise the minimum anyway.
This could be fine if we accept that small risk and it keeps the
testing env from getting crazy; I don't actually want to test vs N
Python versions anyway.

On Wed, Apr 10, 2019 at 4:49 PM shane knapp <sknapp@berkeley.edu> wrote:
>
> ok, i'll create a PR for each branch so we can test things that way, rather than backporting.
>
> the bigger question i have is how far back, branch-wise, are we willing to support w/regards
to tests?
>
> On Wed, Apr 10, 2019 at 12:16 PM Sean Owen <srowen@gmail.com> wrote:
>>
>> In theory Spark 2.4 supports Python 3.4; would this mean it's now just
>> tested vs 3.6? that's not out of the question, but can the older
>> branches continue to test on older versions or is that super complex?
>>
>> On Wed, Apr 10, 2019 at 1:37 PM shane knapp <sknapp@berkeley.edu> wrote:
>> >
>> > details here (see most recent comments for current state of things):
>> > https://issues.apache.org/jira/browse/SPARK-25079
>> >
>> > my PR for these changes:
>> > https://github.com/apache/spark/pull/24266
>> >
>> > we're doing this because we need to support arrow 0.12.1:
>> > https://issues.apache.org/jira/browse/SPARK-27276
>> >
>> > a couple of things:
>> >
>> > * i won't be switching things over until april 11th or 12th at the earliest.
>> > *  this change will require a short (~15-20min) downtime to switch over from
3.4 to 3.6.
>> > *  updating python will impact all active branches, so i will need to backport
this pr:  https://github.com/apache/spark/pull/24266
>> >
>> > question:  which other branches should i be applying this change to?  definitely
master and 2.4...  also 2.3?  2.2?
>> >
>> > thanks in advance,
>> >
>> > shane
>> > --
>> > Shane Knapp
>> > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message