spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shane knapp <skn...@berkeley.edu>
Subject Re: [build system] important: potential upgrading of the python build environment
Date Fri, 07 Jul 2017 15:52:45 GMT
this is done.  i'll babysit builds today.

On Fri, Jul 7, 2017 at 8:40 AM, shane knapp <sknapp@berkeley.edu> wrote:
> doing this now.
>
> On Thu, Jul 6, 2017 at 1:34 PM, shane knapp <sknapp@berkeley.edu> wrote:
>> (big CC list so the people involved have visibility)
>>
>> we're currently using a (very old) installation of anaconda python to
>> manage the python3 build deps, but it turns out that our (very old)
>> versions of numpy and pandas are starting to hold us back.
>>
>> reference:
>> https://github.com/apache/spark/pull/18459
>>
>> i'm proposing an upgrade to bring numpy and pandas up to 1.11.3 and
>> 0.19.2 respectively...  however, as with all things "dependency hell"
>> related, a lot more will be upgraded in the process:
>> https://github.com/apache/spark/pull/18459#issuecomment-313468679
>>
>> the upgrade is easy to do, and to facilitate "holy shit i broke the
>> world" rollbacks, the original anaconda installation has been backed
>> up and i can revert in moments.
>>
>> i'm reasonably sure that this upgrade won't horribly break things, but
>> it's hard to know for sure w/o a dedicated staging environment (also
>> on the books for the end-of-summer new jenkins rollout).
>>
>> if people seem ok w/this plan (update, watch builds for failures,
>> rollback immediately if necessary), i will do this first thing
>> tomorrow morning (~7am PDT).
>>
>> thoughts?  comments?  concerns?
>>
>> shane

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message