spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shane knapp <skn...@berkeley.edu>
Subject Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]
Date Tue, 02 Apr 2019 17:29:51 GMT
i am totally fine w/waiting a few days for the latest arrow release...  not
at all a problem.

On Tue, Apr 2, 2019 at 9:14 AM Bryan Cutler <cutlerb@gmail.com> wrote:

> Nice work Shane! That all sounds good to me.  We might want to use pyarrow
> 0.12.1 though, there is a major bug that was fixed, but we can discuss in
> the PR.  I will put up the code changes in the next few days.
> Felix, I think you're right about Python 3.5, they just list one upcoming
> release and that's not necessarily the last. Comparing the histories, it
> might still be soon though. I think using 3.6 will be fine, as a point of
> reference, pyarrow CI uses 2.7 and 3.6.
>
> On Mon, Apr 1, 2019 at 3:09 PM shane knapp <sknapp@berkeley.edu> wrote:
>
>> well now!  color me completely surprised...  i decided to whip up a fresh
>> python3.6.8 conda environment this morning to "see if things just worked".
>>
>> well, apparently they do!  :)
>>
>> regardless, this is pretty awesome news as i will be able to easily
>> update the 'py3k' python3.4 environment to a fresh, less bloated, but still
>> package-complete python3.6.8 environment (including pyarrow 0.12.0, pandas
>> 0.24.2, scipy 1.2.1).
>>
>> i tested this pretty extensively today on both the ubuntu and centos
>> workers, and i think i'm ready to pull the trigger for a build-system-wide
>> upgrade...   however, i'll be out wednesday through friday this week and
>> don't want to make a massive change before disappearing for a few days.
>>
>> so:  how does early next week sound for the python upgrade?  :)
>>
>> shane
>>
>> On Mon, Apr 1, 2019 at 8:58 AM shane knapp <sknapp@berkeley.edu> wrote:
>>
>>> i'd much prefer that we minimize the number of python versions that we
>>> test against...  would 2.7 and 3.6 be sufficient?
>>>
>>> On Fri, Mar 29, 2019 at 10:23 PM Felix Cheung <felixcheung_m@hotmail.com>
>>> wrote:
>>>
>>>> I don’t take it as Sept 2019 is end of life for python 3.5 tho. It’s
>>>> just saying the next release.
>>>>
>>>> In any case I think in the next release it will be great to get more
>>>> Python 3.x release test coverage.
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* shane knapp <sknapp@berkeley.edu>
>>>> *Sent:* Friday, March 29, 2019 4:46 PM
>>>> *To:* Bryan Cutler
>>>> *Cc:* Felix Cheung; Hyukjin Kwon; dev
>>>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x
>>>> [SPARK-27276]
>>>>
>>>> i'm not opposed to 3.6 at all.
>>>>
>>>> On Fri, Mar 29, 2019 at 4:16 PM Bryan Cutler <cutlerb@gmail.com> wrote:
>>>>
>>>>> PyArrow dropping Python 3.4 was mainly due to support going away at
>>>>> Conda-Forge and other dependencies also dropping it.  I think we better
>>>>> upgrade Jenkins Python while we are at it.  Are you all against jumping
to
>>>>> Python 3.6 so we are not in the same boat in September?
>>>>>
>>>>> On Thu, Mar 28, 2019 at 7:58 PM Felix Cheung <
>>>>> felixcheung_m@hotmail.com> wrote:
>>>>>
>>>>>> 3.4 is end of life but 3.5 is not. From your link
>>>>>>
>>>>>> we expect to release Python 3.5.8 around September 2019.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> *From:* shane knapp <sknapp@berkeley.edu>
>>>>>> *Sent:* Thursday, March 28, 2019 7:54 PM
>>>>>> *To:* Hyukjin Kwon
>>>>>> *Cc:* Bryan Cutler; dev; Felix Cheung
>>>>>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x
>>>>>> [SPARK-27276]
>>>>>>
>>>>>> looks like the same for 3.5...
>>>>>> https://www.python.org/dev/peps/pep-0478/
>>>>>>
>>>>>> let's pick a python version and start testing.
>>>>>>
>>>>>> On Thu, Mar 28, 2019 at 7:52 PM shane knapp <sknapp@berkeley.edu>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>> If there was, it looks inevitable to upgrade Jenkins\s Python
from
>>>>>>>> 3.4 to 3.5.
>>>>>>>>
>>>>>>>> this is inevitable.  3.4s final release was 10 days ago (
>>>>>>> https://www.python.org/dev/peps/pep-0429/) so we're basically
EOL.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Shane Knapp
>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>>> https://rise.cs.berkeley.edu
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Shane Knapp
>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>> https://rise.cs.berkeley.edu
>>>>
>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>

-- 
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Mime
View raw message