spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Liochon <nkey...@gmail.com>
Subject Re: Unit tests in < 5 minutes
Date Fri, 08 Aug 2014 17:19:47 GMT
fwiw, when we did this work in HBase, we categorized the tests. Then some
tests can share a single jvm, while some others need to be isolated in
their own jvm. Nevertheless surefire can still run them in parallel by
starting/stopping several jvm.

Nicolas


On Fri, Aug 8, 2014 at 7:10 PM, Reynold Xin <rxin@databricks.com> wrote:

> ScalaTest actually has support for parallelization built-in. We can use
> that.
>
> The main challenge is to make sure all the test suites can work in parallel
> when running along side each other.
>
>
> On Fri, Aug 8, 2014 at 9:47 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > How about using parallel execution feature of maven-surefire-plugin
> > (assuming all the tests were made parallel friendly) ?
> >
> >
> >
> http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html
> >
> > Cheers
> >
> >
> > On Fri, Aug 8, 2014 at 9:14 AM, Sean Owen <sowen@cloudera.com> wrote:
> >
> > > A common approach is to separate unit tests from integration tests.
> > > Maven has support for this distinction. I'm not sure it helps a lot
> > > though, since it only helps you to not run integration tests all the
> > > time. But lots of Spark tests are integration-test-like and are
> > > important to run to know a change works.
> > >
> > > I haven't heard of a plugin to run different test suites remotely on
> > > many machines, but I would not be surprised if it exists.
> > >
> > > The Jenkins servers aren't CPU-bound as far as I can tell. It's that
> > > the tests spend a lot of time waiting for bits to start up or
> > > complete. That implies the existing tests could be sped up by just
> > > running in parallel locally. I recall someone recently proposed this?
> > >
> > > And I think the problem with that is simply that some of the tests
> > > collide with each other, by opening up the same port at the same time
> > > for example. I know that kind of problem is being attacked even right
> > > now. But if all the tests were made parallel friendly, I imagine
> > > parallelism could be enabled and speed up builds greatly without any
> > > remote machines.
> > >
> > >
> > > On Fri, Aug 8, 2014 at 5:01 PM, Nicholas Chammas
> > > <nicholas.chammas@gmail.com> wrote:
> > > > Howdy,
> > > >
> > > > Do we think it's both feasible and worthwhile to invest in getting
> our
> > > unit
> > > > tests to finish in under 5 minutes (or something similarly brief)
> when
> > > run
> > > > by Jenkins?
> > > >
> > > > Unit tests currently seem to take anywhere from 30 min to 2 hours. As
> > > > people add more tests, I imagine this time will only grow. I think it
> > > would
> > > > be better for both contributors and reviewers if they didn't have to
> > wait
> > > > so long for test results; PR reviews would be shorter, if nothing
> else.
> > > >
> > > > I don't know how how this is normally done, but maybe it wouldn't be
> > too
> > > > much work to get a test cycle to feel lighter.
> > > >
> > > > Most unit tests are independent and can be run concurrently, right?
> > Would
> > > > it make sense to build a given patch on many servers at once and send
> > > > disjoint sets of unit tests to each?
> > > >
> > > > I'd be interested in working on something like that if possible (and
> > > > sensible).
> > > >
> > > > Nick
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > > For additional commands, e-mail: dev-help@spark.apache.org
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message