spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <>
Subject Re: Spurious test failures, testing best practices
Date Tue, 02 Dec 2014 23:46:19 GMT
On Tue, Dec 2, 2014 at 3:39 PM, Ryan Williams
<> wrote:
> Marcelo: by my count, there are 19 maven modules in the codebase. I am
> typically only concerned with "core" (and therefore its two dependencies as
> well, `network/{shuffle,common}`).

But you only need to compile the others once. Once you've established
the baseline, you can just compile / test "core" to your heart's
desire. Core tests won't even run until you build the assembly anyway,
since some of them require the assembly to be present.

Also, even if you work in core - I'd say especially if you work in
core - you should still, at some point, compile and test everything
else that depends on it.

So, do this ONCE:

  mvn install -DskipTests

Then do this as many times as you want:

  mvn -pl spark-core_2.10 something

That doesn't seem too bad to me. (Be aware of the "assembly" comment
above, since testing spark-core means you may have to rebuild the
assembly from time to time, if your changes affect those tests.)

> re: Marcelo's comment about "missing the 'spark-parent' project", I saw that
> error message too and tried to ascertain what it could mean. Why would
> `network/shuffle` need something from the parent project?

The "spark-parent" project is the main pom that defines dependencies
and their version, along with lots of build plugins and
configurations. It's needed by all modules to compile correctly.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message