spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: spark 1.1.0 save data to hdfs failed
Date Fri, 23 Jan 2015 10:41:12 GMT
So, you should not depend on Hadoop artifacts unless you use them
directly. You should mark Hadoop and Spark deps as provided. Then the
cluster's version is used at runtime with spark-submit. That's the
usual way to do it, which works.

If you need to embed Spark in your app and are running it outside the
cluster for some reason, and you have to embed Hadoop and Spark code
in your app, the version has to match. You should also use mvn
dependency:tree to see all the dependencies coming in. There may be
many sources of a Hadoop dep.

On Fri, Jan 23, 2015 at 1:05 AM, ey-chih chow <eychih@hotmail.com> wrote:
> Thanks.  But after I replace the maven dependence from
>
>                 <dependency>
>                          <groupId>org.apache.hadoop</groupId>
>                          <artifactId>hadoop-client</artifactId>
>                          <version>2.5.0-cdh5.2.0</version>
>                          <scope>provided</scope>
>                          <exclusions>
>                            <exclusion>
>                              <groupId>org.mortbay.jetty</groupId>
>                              <artifactId>servlet-api</artifactId>
>                            </exclusion>
>                            <exclusion>
>                              <groupId>javax.servlet</groupId>
>                              <artifactId>servlet-api</artifactId>
>                            </exclusion>
>                            <exclusion>
>                              <groupId>io.netty</groupId>
>                              <artifactId>netty</artifactId>
>                            </exclusion>
>                          </exclusions>
>                 </dependency>
>
> to
>
>                 <dependency>
>
>                          <groupId>org.apache.hadoop</groupId>
>
>                          <artifactId>hadoop-client</artifactId>
>
>                          <version>1.0.4</version>
>
>                          <scope>provided</scope>
>
>                          <exclusions>
>
>                            <exclusion>
>
>                              <groupId>org.mortbay.jetty</groupId>
>
>                              <artifactId>servlet-api</artifactId>
>
>                            </exclusion>
>
>                            <exclusion>
>
>                              <groupId>javax.servlet</groupId>
>
>                              <artifactId>servlet-api</artifactId>
>
>                            </exclusion>
>
>                            <exclusion>
>
>                              <groupId>io.netty</groupId>
>
>                              <artifactId>netty</artifactId>
>
>                            </exclusion>
>
>                          </exclusions>
>
>                 </dependency>
>
>
> the warning message is still shown up in the namenode log.  Is there any
> other thing I need to do?
>
>
> Thanks.
>
>
> Ey-Chih Chow
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message