spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Current way to include hive in a build
Date Fri, 18 Jul 2014 05:57:56 GMT
Hey Stephen,

The only change the build was that we ask users to run -Phive and
-Pyarn of --with-hive and --with-yarn (which internally just set
-Phive and -Pyarn). I don't think this should affect the dependency
graph.

Just to test this, what happens if you run *without* the CDH profile
and build with hadoop version 2.3.0? Does that work?

- Patrick

On Thu, Jul 17, 2014 at 4:00 PM, Stephen Boesch <javadba@gmail.com> wrote:
> Having looked at trunk make-distribution.sh the --with-hive and --with-yarn
> are now deprecated.
>
> Here is the way I have built it:
>
> Added to pom.xml:
>
>    <profile>
>       <id>cdh5</id>
>       <activation>
>         <activeByDefault>false</activeByDefault>
>       </activation>
>       <properties>
>         <hadoop.version>2.3.0-cdh5.0.0</hadoop.version>
>         <yarn.version>2.3.0-cdh5.0.0</yarn.version>
>         <hbase.version>0.96.1.1-cdh5.0.0</hbase.version>
>         <zookeeper.version>3.4.5-cdh5.0.0</zookeeper.version>
>       </properties>
>     </profile>
>
> *mvn -Pyarn -Pcdh5 -Phive -Dhadoop.version=2.3.0-cdh5.0.1
> -Dyarn.version=2.3.0-cdh5.0.0 -DskipTests clean package*
>
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Spark Project Parent POM .......................... SUCCESS [3.165s]
> [INFO] Spark Project Core ................................ SUCCESS
> [2:39.504s]
> [INFO] Spark Project Bagel ............................... SUCCESS [7.596s]
> [INFO] Spark Project GraphX .............................. SUCCESS [22.027s]
> [INFO] Spark Project ML Library .......................... SUCCESS [36.284s]
> [INFO] Spark Project Streaming ........................... SUCCESS [24.309s]
> [INFO] Spark Project Tools ............................... SUCCESS [3.147s]
> [INFO] Spark Project Catalyst ............................ SUCCESS [20.148s]
> [INFO] Spark Project SQL ................................. SUCCESS [18.560s]
> *[INFO] Spark Project Hive ................................ FAILURE
> [33.962s]*
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-dependency-plugin:2.4:copy-dependencies
> (copy-dependencies) on project spark-hive_2.10: Execution copy-dependencies
> of goal
> org.apache.maven.plugins:maven-dependency-plugin:2.4:copy-dependencies
> failed: Plugin org.apache.maven.plugins:maven-dependency-plugin:2.4 or one
> of its dependencies could not be resolved: Could not find artifact
> commons-logging:commons-logging:jar:1.0.4 -> [Help 1]
>
> Anyone who is presently building with -Phive and has a suggestion for this?

Mime
View raw message