spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Scala 2.11 builds broken/ Can the PR build run also 2.11?
Date Mon, 12 Oct 2015 16:25:49 GMT
It's really easy to create and modify those builds. If the issue is that we
need to add SBT or Maven to the existing one, it's a short change. We can
just have it build both of them. I wasn't aware of things breaking before
in one build but not another.

- Patrick

On Mon, Oct 12, 2015 at 9:21 AM, Sean Owen <sowen@cloudera.com> wrote:

> Yeah, was the issue that it had to be built vs Maven to show the error
> and this uses SBT -- or vice versa? that's why the existing test
> didn't detect it. Was just thinking of adding one more of these non-PR
> builds, but I forget if there was a reason this is hard. Certainly not
> worth building for each PR.
>
> On Mon, Oct 12, 2015 at 5:16 PM, Patrick Wendell <pwendell@gmail.com>
> wrote:
> > We already do automated compile testing for Scala 2.11 similar to Hadoop
> > versions:
> >
> > https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/
> >
> https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/job/Spark-master-Scala211-Compile/buildTimeTrend
> >
> >
> > If you look, this build takes 7-10 minutes, so it's a nontrivial
> increase to
> > add it to all new PR's. Also, it's only broken once in the last few
> months
> > (despite many patches going in) - a pretty low failure rate. For
> scenarios
> > like this it's better to test it asynchronously. We can even just revert
> a
> > patch immediately if it's found to break 2.11.
> >
> > Put another way - we typically have 1000 patches or more per release.
> Even
> > at one jenkins run per patch: 7 minutes * 1000 = 7 days of developer
> > productivity loss. Compare that to having a few times where we have to
> > revert a patch and ask someone to resubmit (which maybe takes at most one
> > hour)... it's not worth it.
> >
> > - Patrick
> >
> > On Mon, Oct 12, 2015 at 8:24 AM, Sean Owen <sowen@cloudera.com> wrote:
> >>
> >> There are many Jenkins jobs besides the pull request builder that
> >> build against various Hadoop combinations, for example, in the
> >> background. Is there an obstacle to building vs 2.11 on both Maven and
> >> SBT this way?
> >>
> >> On Mon, Oct 12, 2015 at 2:55 PM, Iulian DragoČ™
> >> <iulian.dragos@typesafe.com> wrote:
> >> > Anything that can be done by a machine should be done by a machine. I
> am
> >> > not
> >> > sure we have enough data to say it's only once or twice per release,
> and
> >> > even if we were to issue a PR for each breakage, it's additional load
> on
> >> > committers and reviewers, not to mention our own work. I personally
> >> > don't
> >> > see how 2-3 minutes of compute time per PR can justify hours of work
> >> > plus
> >> > reviews.
> >
> >
>

Mime
View raw message