spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Cutler <cutl...@gmail.com>
Subject Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]
Date Tue, 02 Apr 2019 16:14:27 GMT
Nice work Shane! That all sounds good to me.  We might want to use pyarrow
0.12.1 though, there is a major bug that was fixed, but we can discuss in
the PR.  I will put up the code changes in the next few days.
Felix, I think you're right about Python 3.5, they just list one upcoming
release and that's not necessarily the last. Comparing the histories, it
might still be soon though. I think using 3.6 will be fine, as a point of
reference, pyarrow CI uses 2.7 and 3.6.

On Mon, Apr 1, 2019 at 3:09 PM shane knapp <sknapp@berkeley.edu> wrote:

> well now!  color me completely surprised...  i decided to whip up a fresh
> python3.6.8 conda environment this morning to "see if things just worked".
>
> well, apparently they do!  :)
>
> regardless, this is pretty awesome news as i will be able to easily update
> the 'py3k' python3.4 environment to a fresh, less bloated, but still
> package-complete python3.6.8 environment (including pyarrow 0.12.0, pandas
> 0.24.2, scipy 1.2.1).
>
> i tested this pretty extensively today on both the ubuntu and centos
> workers, and i think i'm ready to pull the trigger for a build-system-wide
> upgrade...   however, i'll be out wednesday through friday this week and
> don't want to make a massive change before disappearing for a few days.
>
> so:  how does early next week sound for the python upgrade?  :)
>
> shane
>
> On Mon, Apr 1, 2019 at 8:58 AM shane knapp <sknapp@berkeley.edu> wrote:
>
>> i'd much prefer that we minimize the number of python versions that we
>> test against...  would 2.7 and 3.6 be sufficient?
>>
>> On Fri, Mar 29, 2019 at 10:23 PM Felix Cheung <felixcheung_m@hotmail.com>
>> wrote:
>>
>>> I don’t take it as Sept 2019 is end of life for python 3.5 tho. It’s
>>> just saying the next release.
>>>
>>> In any case I think in the next release it will be great to get more
>>> Python 3.x release test coverage.
>>>
>>>
>>>
>>> ------------------------------
>>> *From:* shane knapp <sknapp@berkeley.edu>
>>> *Sent:* Friday, March 29, 2019 4:46 PM
>>> *To:* Bryan Cutler
>>> *Cc:* Felix Cheung; Hyukjin Kwon; dev
>>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x
>>> [SPARK-27276]
>>>
>>> i'm not opposed to 3.6 at all.
>>>
>>> On Fri, Mar 29, 2019 at 4:16 PM Bryan Cutler <cutlerb@gmail.com> wrote:
>>>
>>>> PyArrow dropping Python 3.4 was mainly due to support going away at
>>>> Conda-Forge and other dependencies also dropping it.  I think we better
>>>> upgrade Jenkins Python while we are at it.  Are you all against jumping to
>>>> Python 3.6 so we are not in the same boat in September?
>>>>
>>>> On Thu, Mar 28, 2019 at 7:58 PM Felix Cheung <felixcheung_m@hotmail.com>
>>>> wrote:
>>>>
>>>>> 3.4 is end of life but 3.5 is not. From your link
>>>>>
>>>>> we expect to release Python 3.5.8 around September 2019.
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> *From:* shane knapp <sknapp@berkeley.edu>
>>>>> *Sent:* Thursday, March 28, 2019 7:54 PM
>>>>> *To:* Hyukjin Kwon
>>>>> *Cc:* Bryan Cutler; dev; Felix Cheung
>>>>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x
>>>>> [SPARK-27276]
>>>>>
>>>>> looks like the same for 3.5...
>>>>> https://www.python.org/dev/peps/pep-0478/
>>>>>
>>>>> let's pick a python version and start testing.
>>>>>
>>>>> On Thu, Mar 28, 2019 at 7:52 PM shane knapp <sknapp@berkeley.edu>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>> If there was, it looks inevitable to upgrade Jenkins\s Python
from
>>>>>>> 3.4 to 3.5.
>>>>>>>
>>>>>>> this is inevitable.  3.4s final release was 10 days ago (
>>>>>> https://www.python.org/dev/peps/pep-0429/) so we're basically EOL.
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Shane Knapp
>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>> https://rise.cs.berkeley.edu
>>>>>
>>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>

Mime
View raw message