spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (Jira)" <>
Subject [jira] [Commented] (SPARK-29329) maven incremental builds not working
Date Mon, 21 Oct 2019 18:27:00 GMT


Thomas Graves commented on SPARK-29329:

from the linked issue above it sounds like the zinc compile server mode isn't really supported
anymore in scala maven plugin 4.x. the zinc version changed to 1 and the code is completely
different and sounds like they only support the embedded mode and zinc server option isn't
available so if that is the case it isn't helping us and it recompiles every time.

There are still a few things unclear to me from the issue above in the wording in their response.
I haven't had time to look at it more since their response.  Sounds like someone working
on "bloop" compiler server replacement for scala.

> maven incremental builds not working
> ------------------------------------
>                 Key: SPARK-29329
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Build
>    Affects Versions: 3.0.0
>            Reporter: Thomas Graves
>            Priority: Major
> It looks like since we Upgraded scala-maven-plugin to 4.2.0
spark incremental builds stop working.  Everytime you build its building all files, which
takes forever.
> It would be nice to fix this.
> To reproduce, just build spark once ( I happened to be using the command below):
> build/mvn -Phadoop-3.2 -Phive-thriftserver -Phive -Pyarn -Pkinesis-asl -Pkubernetes -Pmesos
-Phadoop-cloud -Pspark-ganglia-lgpl package -DskipTests
> Then build it again and you will see that it compiles all the files and takes 15-30 minutes.
With incremental it skips all unnecessary files and takes closer to 5 minutes.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message