spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: Hadoop interface vs class
Date Thu, 26 Jun 2014 19:28:14 GMT
On Thu, Jun 26, 2014 at 1:44 PM, Robert James <> wrote:
> I downloaded the Spark distro for Hadoop 2, and installed it on my
> machine.  But the code doesn't have a reference to that path - it uses
> sbt for dependencies.  As far as I can tell, using sbt or maven or ivy
> will always result in a transitive dependency to Hadoop 1 ????

That's what happens now, yes. There is one set of artifacts out there,
and it is the Hadoop 1 version. (FWIW I also think it would help to
have Hadoop 2-flavored artifacts published too.)

> Shouldn't there be a maven option to use Hadoop 2, just like there is
> a different package to download?

You mean in your dependencies? yeah, you override the hadoop-client dependency.

View raw message