spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shane knapp <skn...@berkeley.edu>
Subject Re: [build system] important: potential upgrading of the python build environment
Date Fri, 07 Jul 2017 15:40:50 GMT
doing this now.

On Thu, Jul 6, 2017 at 1:34 PM, shane knapp <sknapp@berkeley.edu> wrote:
> (big CC list so the people involved have visibility)
>
> we're currently using a (very old) installation of anaconda python to
> manage the python3 build deps, but it turns out that our (very old)
> versions of numpy and pandas are starting to hold us back.
>
> reference:
> https://github.com/apache/spark/pull/18459
>
> i'm proposing an upgrade to bring numpy and pandas up to 1.11.3 and
> 0.19.2 respectively...  however, as with all things "dependency hell"
> related, a lot more will be upgraded in the process:
> https://github.com/apache/spark/pull/18459#issuecomment-313468679
>
> the upgrade is easy to do, and to facilitate "holy shit i broke the
> world" rollbacks, the original anaconda installation has been backed
> up and i can revert in moments.
>
> i'm reasonably sure that this upgrade won't horribly break things, but
> it's hard to know for sure w/o a dedicated staging environment (also
> on the books for the end-of-summer new jenkins rollout).
>
> if people seem ok w/this plan (update, watch builds for failures,
> rollback immediately if necessary), i will do this first thing
> tomorrow morning (~7am PDT).
>
> thoughts?  comments?  concerns?
>
> shane

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message