spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mildebrandt <ch...@woodenrhino.com>
Subject Problems building Spark for Hadoop 1.0.3
Date Mon, 13 Oct 2014 21:39:48 GMT
Hello,

After reading the following pages:
https://spark.apache.org/docs/latest/building-with-maven.html
http://spark.apache.org/docs/latest/hadoop-third-party-distributions.html

I came up with this build command:
mvn -Dhadoop.version=1.0.3 -DskipTests -Pdeb -Phadoop-1.0.3 -Phive
-Phadoop-provided clean package

This fails with the following error:
[WARNING] The requested profile "hadoop-1.0.3" could not be activated
because it does not exist.
[ERROR] Failed to execute goal on project spark-parent: Could not resolve
dependencies for project org.apache.spark:spark-parent:pom:1.1.0: The
following artifacts could not be resolved:
org.apache.hadoop:hadoop-yarn-api:jar:1.0.3,
org.apache.hadoop:hadoop-yarn-common:jar:1.0.3,
org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:1.0.3,
org.apache.hadoop:hadoop-yarn-client:jar:1.0.3: Failure to find
org.apache.hadoop:hadoop-yarn-api:jar:1.0.3 in
https://repo1.maven.org/maven2 was cached in the local repository,
resolution will not be reattempted until the update interval of central has
elapsed or updates are forced -> [Help 1]

If I remove the hadoop-provided profile, it works fine. So my questions:
 - Why isn't there a hadoop-1.0.3 profile? The
hadoop-third-party-distributions.html page seems to suggest that it would
exist.
 - It's true, I don't see a hadoop-yarn-api:jar:1.0.3 in the maven
repository. I'd like to build without yarn support. And ideas why it's
pulling yarn? Is it due to the fact that the hadoop-1.0.3 profile doesn't
exist?
 - Since running without the hadoop-provided profile works, is there any
issue running that version of Spark against my Apache 1.0.3 cluster?

Thanks,
-Chris



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problems-building-Spark-for-Hadoop-1-0-3-tp16324.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message