spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-3431) Parallelize execution of tests
Date Mon, 08 Dec 2014 23:00:13 GMT


Sean Owen commented on SPARK-3431:

Surefire is definitely the main Maven testing plugin and has all the bells and whistles. scalatest
is a fork of a quite old verison.

Well, in theory I think these are all the things that need to happen:

- Make sure that the test-compile phase compiles all of the Scala-based tests as well as Java-based
tests. I am pretty sure this happens correctly already from the Maven Scala plugin.
- Port the scalatest config to the surefire plugin. I bet it all Just Works given that scalatest
is derived from surefire.
- Delete scalatest config
- Un-disable the surefire config
- Probably add config to make sure "<includes>" includes all of the names of all Java
and Scala tests

Then you get a lot of parallelization options for sure.

Off the top of my head it should work, but then again, maybe there was a good reason surefire
was never used.

> Parallelize execution of tests
> ------------------------------
>                 Key: SPARK-3431
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Build
>            Reporter: Nicholas Chammas
>            Assignee: Nicholas Chammas
> Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common strategy to
cut test time down is to parallelize the execution of the tests. Doing that may in turn require
some prerequisite changes to be made to how certain tests run.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message