spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Difference bewteen library dependencies version
Date Thu, 04 Jun 2015 13:42:37 GMT
For your first question, please take a look at HADOOP-9922.
The fix is in hadoop-common module.

Cheers

On Thu, Jun 4, 2015 at 2:53 AM, Jean-Charles RISCH <
risch.jeancharles@gmail.com> wrote:

> Hello,
> *(Before everything : I use IntellijIdea 14.0.1, SBT and Scala 2.11.6)*
>
> This morning, I was looking to resolve the "Failed to locate the winutils
> binary in the hadoop binary path" error.
>
>
> I noticed that I can solve it configuring my build.sbt to
>
> "...
>
> libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "1.0.4"
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.1" excludeAll(
>   ExclusionRule(organization = "org.apache.hadoop")
>   )
>
> libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.3.1" excludeAll(
>   ExclusionRule(organization = "org.apache.hadoop")
>   )
>
> "
>
>
> but if i change the line
>
> libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "1.0.4"
>
> to
>
> libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.7.0"
>
> the error is back.
>
>
> What does it mean? Spark is build for an old version of hadoop? I really
> want to understand.
>
> *Also, a bonus question : *
> As you can see I am using spark 1.3.1 and spark-mllib APIs. I am using the
> last version, but my APIs are not corresponding to the latest official APIs
> (https://spark.apache.org/docs/*latest*/api/scala/#package)
>
> For example, to run a KMeans algo, I have to use KMeans.train() whereas it
> does not exist in the latest API.
>
> First time, I ask something in the mailing list, I hope I use it well.
> Sorry for my bad english.
>
> Thank you and have a good day,
>
> JC
> ᐧ
>

Mime
View raw message