spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shane knapp <skn...@berkeley.edu>
Subject Re: JENKINS: downtime next week, wed and thurs mornings (9-23 and 9-24)
Date Tue, 22 Sep 2015 17:07:56 GMT
ok, here's the updated downtime schedule for this week:

wednesday, sept 23rd:

firewall maintenance cancelled, as jon took care of the update
saturday morning while we were bringing jenkins back up after the colo
fire

thursday, sept 24th:

jenkins maintenance is still scheduled, but abbreviated as some of the
maintenance was performed saturday morning as well
* new builds will stop being accepted ~630am PDT
  - i'll kill any hangers-on at 730am, and after maintenance is done,
i will retrigger any killed jobs
* jenkins worker system package updates
  - amp-jenkins-master was completed on saturday
  - this will NOT include kernel updates as moving to
2.6.32-573.3.1.el6 bricked amp-jenkins-master
* moving default system java for builds from jdk1.7.0_71 to jdk1.7.0_79
* all systems get a reboot
* expected downtime:  3.5 hours or so

i'll post updates as i progress.

also, i'll post a copy of our post-mortem once the dust settles.  it's
been, shall we say, a pretty crazy few days.

http://news.berkeley.edu/2015/09/19/campus-network-outage/

:)

On Mon, Sep 21, 2015 at 10:11 AM, shane knapp <sknapp@berkeley.edu> wrote:
> quick update:  we actually did some of the maintenance on our systems
> after the berkeley-wide outage caused by one of our (non-jenkins)
> servers halting and catching fire.
>
> we'll still have some downtime early wednesday, but tomorrow's will be
> cancelled.  i'll send out another update real soon now with what we'll
> be covering on wednesday once we get our current situation more under
> control.  :)
>
> On Wed, Sep 16, 2015 at 12:15 PM, shane knapp <sknapp@berkeley.edu> wrote:
>>> 630am-10am thursday, 9-24-15:
>>> * jenknins update to 1.629 (we're a few months behind in versions, and
>>> some big bugs have been fixed)
>>> * jenkins master and worker system package updates
>>> * all systems get a reboot (lots of hanging java processes have been
>>> building up over the months)
>>> * builds will stop being accepted ~630am, and i'll kill any hangers-on
>>> at 730am, and retrigger once we're done
>>> * expected downtime:  3.5 hours or so
>>> * i will also be testing out some of my shiny new ansible playbooks
>>> for the system updates!
>>>
>> i forgot one thing:
>>
>> * moving default system java for builds from jdk1.7.0_71 to jdk1.7.0_79

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message