mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gokhan Capan <gkhn...@gmail.com>
Subject Re: Mahout and Hadoop 2.2.0
Date Wed, 11 Dec 2013 20:09:09 GMT
I apologize, Suneel is right, Counter breaks the binary compatibility.

Well, I can say there is a work in progress for building mahout against
hadoop2.

Gokhan


On Wed, Dec 11, 2013 at 10:03 PM, Hi There <srudamaster@yahoo.com> wrote:

> Here are the full contents of my pom file:
>
> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance"
>   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd">
>   <modelVersion>4.0.0</modelVersion>
>
>   <groupId>clustertest</groupId>
>   <artifactId>clustertest</artifactId>
>   <version>1.0</version>
>   <packaging>jar</packaging>
>
>   <name>clustertest</name>
>   <url>http://maven.apache.org</url>
>     <build>
>         <plugins>
>             <plugin>
>                 <groupId>org.apache.maven.plugins</groupId>
>                 <artifactId>maven-compiler-plugin</artifactId>
>                 <version>2.3.2</version>
>                 <configuration>
>                     <source>1.7</source>
>                     <target>1.7</target>
>                 </configuration>
>             </plugin>
>         </plugins>
>     </build>
>     <properties>
>     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
>   </properties>
>
>   <dependencies>
>     <dependency>
>       <groupId>junit</groupId>
>       <artifactId>junit</artifactId>
>       <version>3.8.1</version>
>       <scope>test</scope>
>     </dependency>
>     <dependency>
>       <groupId>org.apache.mahout</groupId>
>       <artifactId>mahout-core</artifactId>
>       <version>0.8</version>
>       <type>jar</type>
>       <exclusions>
>         <exclusion>
>           <artifactId>hadoop-core</artifactId>
>           <groupId>org.apache.hadoop</groupId>
>         </exclusion>
>       </exclusions>
>     </dependency>
>     <dependency>
>       <groupId>com.google.code.gson</groupId>
>       <artifactId>gson</artifactId>
>       <version>2.2.4</version>
>       <type>jar</type>
>     </dependency>
>     <dependency>
>       <groupId>org.apache.mahout</groupId>
>       <artifactId>mahout-integration</artifactId>
>       <version>0.8</version>
>       <type>jar</type>
>       <exclusions>
>         <exclusion>
>           <artifactId>hadoop-core</artifactId>
>           <groupId>org.apache.hadoop</groupId>
>         </exclusion>
>       </exclusions>
>     </dependency>
>     <dependency>
>         <groupId>org.apache.hadoop</groupId>
>         <artifactId>hadoop-client</artifactId>
>         <version>2.2.0</version>
>     </dependency>
>   </dependencies>
> </project>
>
> How/where do I check to see whether my hadoop cluster is 2.2.0? Sorry, I
> am new at this.
>
>
>
>
> On Wednesday, December 11, 2013 11:56 AM, Gokhan Capan <gkhncpn@gmail.com>
> wrote:
>
> Could you check the following?
>
> Are you sure that your hadoop cluster is hadoop 2.2.0?
> Are you sure other dependencies of your project do not have a transitive
> dependency to hadoop?
>
> Gokhan
>
>
>
> On Wed, Dec 11, 2013 at 9:46 PM, Hi There <srudamaster@yahoo.com> wrote:
>
> > I tried to run SparseVectorsFromSequenceFiles, specifying a directory
> with
> > sequence files, and I got the following error:
> >
> > java.lang.Exception: java.lang.IncompatibleClassChangeError: Found
> > interface org.apache.hadoop.mapreduce.Counter, but class was expected
> >
> > Here is a relevant snippet of my pom file for maven:
> >
> > <dependency>
> >       <groupId>org.apache.mahout</groupId>
> >       <artifactId>mahout-core</artifactId>
> >       <version>0.8</version>
> >       <type>jar</type>
> >       <exclusions>
> >         <exclusion>
> >           <artifactId>hadoop-core</artifactId>
> >           <groupId>org.apache.hadoop</groupId>
> >         </exclusion>
> >       </exclusions>
> > </dependency>
> > <dependency>
> >       <groupId>org.apache.mahout</groupId>
> >       <artifactId>mahout-integration</artifactId>
> >       <version>0.8</version>
> >       <type>jar</type>
> >       <exclusions>
> >         <exclusion>
> >           <artifactId>hadoop-core</artifactId>
> >           <groupId>org.apache.hadoop</groupId>
> >         </exclusion>
> >       </exclusions>
> > </dependency>
> > <dependency>
> >         <groupId>org.apache.hadoop</groupId>
> >         <artifactId>hadoop-client</artifactId>
> >         <version>2.2.0</version>
> > </dependency>
> >
> >
> > How do I change this to use the correct version(s) of mahout/hadoop?
> >
> > Thanks!
> >
> >
> >
> >
> > On Tuesday, December 10, 2013 1:03 PM, Gokhan Capan <gkhncpn@gmail.com>
> > wrote:
> >
> > I meant that you shouldn't need to modify mahout's dependencies, just mvn
> > package and it should work against hadoop 2.2.0 (Yeah, 2.2.0 is not
> alpha)
> >
> > Quoting from
> >
> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> > "First, we ensure binary compatibility to the applications that use old
> > mapred APIs. This means that applications which were built against MRv1
> > mapred APIs can run directly on YARN without recompilation, merely by
> > pointing them to an Apache Hadoop 2.x cluster via configuration."
> >
> > If you encounter with any problems, just let the list know.
> >
> > Best
> >
> >
> >
> > > On Dec 9, 2013, at 9:40 PM, Hi There <srudamaster@yahoo.com> wrote:
> > >
> > > Hi Gokhan,
> > >
> > > My project currently fetches every dependency through Maven--is there
> > any way I can grab the version you mentioned that way?
> > >
> > > In that vein, I am using the following version of hadoop:
> > > <dependency>
> > >        <groupId>org.apache.hadoop</groupId>
> > >        <artifactId>hadoop-client</artifactId>
> > >        <version>2.2.0</version>
> > > </dependency>
> > >
> > >
> > > That's not alpha, right?
> > >
> > > Thanks!
> > >
> > >
> > >
> > >
> > >
> > > On Monday, December 9, 2013 10:05 AM, Gokhan Capan <gkhncpn@gmail.com>
> > wrote:
> > >
> > > Mahout actually should work with hadoop-2 stable without recompiling,
> > > not with hadoop-2 alpha though.
> > >
> > > We're, by the way, currently in the process of adding support to build
> > > mahout with hadoop-2.
> > >
> > > Please see mahout-1354 for the relevant issue
> > >
> > > Sent from my iPhone
> > >
> > >
> > >> On Dec 9, 2013, at 19:54, Hi There <srudamaster@yahoo.com> wrote:
> > >>
> > >> Is Dec 2013 still the intended release date of the next mahout release
> > that will be compatible with Hadoop 2.2.0?
> > >>
> > >>
> > >>
> > >>
> > >> On Thursday, November 21, 2013 12:36 PM, Suneel Marthi <
> > suneel_marthi@yahoo.com> wrote:
> > >>
> > >> Targeted for Dec 2013.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Thursday, November 21, 2013 3:26 PM, Hi There <
> srudamaster@yahoo.com>
> > wrote:
> > >>
> > >> Thanks for the reply! Is there a timeline for then the next release
> > will be?
> > >>
> > >>
> > >> Thanks,
> > >> Victor
> > >>
> > >>
> > >>
> > >>
> > >> On Tuesday, November 19, 2013 7:30 PM, Suneel Marthi <
> > suneel_marthi@yahoo.com> wrote:
> > >>
> > >> Hi Victor,
> > >>
> > >> Future releases of Mahout will support Hadoop 2.x, the present
> codebase
> > still only supports Hadoop 1.x.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Tuesday, November 19, 2013 1:42 PM, Hi There <
> srudamaster@yahoo.com>
> > wrote:
> > >>
> > >>
> > >>
> > >> Hello,
> > >>
> > >> I recently upgraded to hadoop's
> > >> newest release, and it seems one of their interfaces has changed, and
> > >> when I try to create sparse vectors from sequence files, I get the
> > >> following exception:
> > >>
> > >> java.lang.IncompatibleClassChangeError: Found interface
> > org.apache.hadoop.mapreduce.Counter, but class was expected
> > >>
> > >> I can include more of the stack trace if necessary.
> > >>
> > >> Are there any plans in the immediate future to upgrade mahout to be
> > compatible with the newest hadoop release?
> > >>
> > >> Thanks,
> > >> Victor
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message