spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shane knapp ☠ <>
Subject [build system] IMPORTANT: builds will be impacted this month
Date Mon, 02 Nov 2020 21:16:31 GMT
TL;DR:  our build system is ancient, EOLed and about to get hit hard w/a
secops hammer.  we need to literally reinstall the entire cluster from
scratch and get things working.

here are the high level bullet points about what's coming up in the next

** all amp-jenkins-worker-* nodes are running centos 6, and the remainder
ubuntu 16.  these will be upgraded to ubuntu 20.

i will be doing this in stages so as to minimize downtime.

** ALL BUILDS NEED TO BE PORTED TO UBUNTU 20.  i can ensure that the
environments on the nodes are identical, but i have yet been able to
successfully build any SBT jobs on any version of ubuntu, and the MVN
builds won't run on ubuntu 18 (tho they work fine on 16).  i also have had
difficulty getting the PRB job to successfully finish on ubuntu.

for this, i will definitely need help from the dev community to get things
working...  and the speed at which things are fixed will be inversely
proportional to how much help i get.  :)

** amplab jenkins primary node will need two major upgrades:  OS from
centos 6 to ubuntu 20, and jenkins from 1.6 to 2.X LTS...

i'm most concerned about this, as it is literally the exact same jenkins
installtion that patrick wendell set up over 10 years ago.  there are many
publish secrets that are entered in to the jenkins config and i'd really
hope that we don't lose them.

my plan here is to upgrade the current jenkins, and fix any things that
break.  then we'll rsync jenkins' homedir to the new primary node and hope
that works.  :)

** user audits

UC berkeley's new security standards require quarterly audits of
non-affiliated accounts...  this won't impact only but a few people on this
list, but i'll need to work w/campus and our department on solutions for
this other than local accounts on the servers.

a LOT is going to happen, and i'm meeting w/my team today and will come up
w/a basic plan.  we will definitely experience downtime during this, but i
cannot guess as to what that will look like.

this might also be a good time to talk about the future of the build
system, auditing our builds (do we need SBT?), or even finally getting
around to dockerizing everything  so i don't need such a fragile and
non-atomic set of worker nodes specifically for spark.

thoughts?  comments?


ps -- this is one of the reasons why i haven't been around much lately...
it's been really tough keeping things up to date while trying to remotely
train up one of my sysadmins to take over some of my build system duties.
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead

View raw message