spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Unit tests in < 5 minutes
Date Fri, 08 Aug 2014 16:47:58 GMT
How about using parallel execution feature of maven-surefire-plugin
(assuming all the tests were made parallel friendly) ?

http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html

Cheers


On Fri, Aug 8, 2014 at 9:14 AM, Sean Owen <sowen@cloudera.com> wrote:

> A common approach is to separate unit tests from integration tests.
> Maven has support for this distinction. I'm not sure it helps a lot
> though, since it only helps you to not run integration tests all the
> time. But lots of Spark tests are integration-test-like and are
> important to run to know a change works.
>
> I haven't heard of a plugin to run different test suites remotely on
> many machines, but I would not be surprised if it exists.
>
> The Jenkins servers aren't CPU-bound as far as I can tell. It's that
> the tests spend a lot of time waiting for bits to start up or
> complete. That implies the existing tests could be sped up by just
> running in parallel locally. I recall someone recently proposed this?
>
> And I think the problem with that is simply that some of the tests
> collide with each other, by opening up the same port at the same time
> for example. I know that kind of problem is being attacked even right
> now. But if all the tests were made parallel friendly, I imagine
> parallelism could be enabled and speed up builds greatly without any
> remote machines.
>
>
> On Fri, Aug 8, 2014 at 5:01 PM, Nicholas Chammas
> <nicholas.chammas@gmail.com> wrote:
> > Howdy,
> >
> > Do we think it's both feasible and worthwhile to invest in getting our
> unit
> > tests to finish in under 5 minutes (or something similarly brief) when
> run
> > by Jenkins?
> >
> > Unit tests currently seem to take anywhere from 30 min to 2 hours. As
> > people add more tests, I imagine this time will only grow. I think it
> would
> > be better for both contributors and reviewers if they didn't have to wait
> > so long for test results; PR reviews would be shorter, if nothing else.
> >
> > I don't know how how this is normally done, but maybe it wouldn't be too
> > much work to get a test cycle to feel lighter.
> >
> > Most unit tests are independent and can be run concurrently, right? Would
> > it make sense to build a given patch on many servers at once and send
> > disjoint sets of unit tests to each?
> >
> > I'd be interested in working on something like that if possible (and
> > sensible).
> >
> > Nick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message