spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chester Chen <ches...@alpinenow.com>
Subject Build spark 1.5.1 branch fails
Date Thu, 08 Oct 2015 17:35:17 GMT
Question regarding branch-1.5  build.

Noticed that the spark project no longer publish the spark-assembly. We
have to build ourselves ( until we find way to not depends on assembly
jar).


I check out the tag v.1.5.1 release version and using the sbt to build it,
I get the following error

build/sbt -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive
-Phive-thriftserver -DskipTests clean package assembly


[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] ::          UNRESOLVED DEPENDENCIES         ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.apache.spark#spark-network-common_2.10;1.5.1: configuration
not public in org.apache.spark#spark-network-common_2.10;1.5.1: 'test'. It
was required from org.apache.spark#spark-network-shuffle_2.10;1.5.1 test
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.apache.spark:spark-network-common_2.10:1.5.1
((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
[warn]  +- org.apache.spark:spark-network-shuffle_2.10:1.5.1
[info] Packaging
/Users/chester/projects/alpine/apache/spark/launcher/target/scala-2.10/spark-launcher_2.10-1.5.1.jar
...
[info] Done packaging.
[warn] four warnings found
[warn] Note: Some input files use unchecked or unsafe operations.
[warn] Note: Recompile with -Xlint:unchecked for details.
[warn] No main class detected
[info] Packaging
/Users/chester/projects/alpine/apache/spark/external/flume-sink/target/scala-2.10/spark-streaming-flume-sink_2.10-1.5.1.jar
...
[info] Done packaging.
sbt.ResolveException: unresolved dependency:
org.apache.spark#spark-network-common_2.10;1.5.1: configuration not public
in org.apache.spark#spark-network-common_2.10;1.5.1: 'test'. It was
required from org.apache.spark#spark-network-shuffle_2.10;1.5.1 test


Somehow the network-shuffle can't find the test jar needed ( not sure why
test still needed, even the  -DskipTests is already specified)

tried the maven command, the build failed as well ( without assembly)

mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver
-DskipTests clean package

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:1.4:enforce
(enforce-versions) on project spark-parent_2.10: Some Enforcer rules have
failed. Look above for specific messages explaining why the rule failed. ->
[Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException



I checkout the branch-1.5 and replaced "1.5.2-SNAPSHOT" with "1.5.1" and
build/sbt will still fail ( same error as above for sbt)

But if I keep the version string as "1.5.2-SNAPSHOT", the build/sbt works
fine.


Any ideas ?

Chester

Mime
View raw message