spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shane knapp <skn...@berkeley.edu>
Subject Re: jenkins httpd being flaky
Date Fri, 13 Mar 2015 20:55:31 GMT
ok, things seem to have stabilized...  httpd hasn't flaked since ~noon, the
hanging PRB job on amp-jenkins-worker-06 was removed w/the restart and
things are now building.

i cancelled and retriggered a bunch of PRB builds, btw:
4848 (https://github.com/apache/spark/pull/3699)
5922 (https://github.com/apache/spark/pull/4733)
5987 (https://github.com/apache/spark/pull/4986)
6222 (https://github.com/apache/spark/pull/4964)
6325 (https://github.com/apache/spark/pull/5018)

as well as:
spark-master-maven-with-yarn

sorry for the inconvenience...  i'm still a little stumped as to what
happened, but i think it was a confluence of events (httpd flaking,
problems at github, mercury in retrograde, friday thinking it's monday).

shane

On Fri, Mar 13, 2015 at 1:08 PM, shane knapp <sknapp@berkeley.edu> wrote:

> i tried a couple of things, but will also be doing a jenkins reboot as
> soon as the current batch of builds finish.
>
>
>
> On Fri, Mar 13, 2015 at 12:40 PM, shane knapp <sknapp@berkeley.edu> wrote:
>
>> ok we have a few different things happening:
>>
>> 1) httpd on the jenkins master is randomly (though not currently) flaking
>> out and causing visits to the site to return a 503.  nothing in the logs
>> shows any problems.
>>
>> 2) there are some github timeouts, which i tracked down and think it's a
>> problem with github themselves (see:  https://status.github.com/ and
>> scroll down to 'mean hook delivery time')
>>
>> 3) we have one spark job w/a strange ivy lock issue, that i just
>> retriggered (https://github.com/apache/spark/pull/4964)
>>
>> 4) there's an errant, unkillable pull request builder job (
>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28574/console
>> )
>>
>> more updates forthcoming.
>>
>> On Fri, Mar 13, 2015 at 12:04 PM, shane knapp <sknapp@berkeley.edu>
>> wrote:
>>
>>> we just started having issues when visiting jenkins and getting 503
>>> service unavailable errors.
>>>
>>> i'm on it and will report back with an all-clear.
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message