mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gokhan Capan <gkhn...@gmail.com>
Subject Re: Mahout and Hadoop 2.2.0
Date Wed, 11 Dec 2013 21:44:48 GMT
In the meantime, you might apply the patch in MAHOUT-1354, build mahout
using mvn package -Phadoop2 -DskipTests=true, use that mahout version and
see if that works

Gokhan


On Wed, Dec 11, 2013 at 10:09 PM, Gokhan Capan <gkhncpn@gmail.com> wrote:

> I apologize, Suneel is right, Counter breaks the binary compatibility.
>
> Well, I can say there is a work in progress for building mahout against
> hadoop2.
>
> Gokhan
>
>
> On Wed, Dec 11, 2013 at 10:03 PM, Hi There <srudamaster@yahoo.com> wrote:
>
>> Here are the full contents of my pom file:
>>
>> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
>> http://www.w3.org/2001/XMLSchema-instance"
>>   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>> http://maven.apache.org/xsd/maven-4.0.0.xsd">
>>   <modelVersion>4.0.0</modelVersion>
>>
>>   <groupId>clustertest</groupId>
>>   <artifactId>clustertest</artifactId>
>>   <version>1.0</version>
>>   <packaging>jar</packaging>
>>
>>   <name>clustertest</name>
>>   <url>http://maven.apache.org</url>
>>     <build>
>>         <plugins>
>>             <plugin>
>>                 <groupId>org.apache.maven.plugins</groupId>
>>                 <artifactId>maven-compiler-plugin</artifactId>
>>                 <version>2.3.2</version>
>>                 <configuration>
>>                     <source>1.7</source>
>>                     <target>1.7</target>
>>                 </configuration>
>>             </plugin>
>>         </plugins>
>>     </build>
>>     <properties>
>>     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
>>   </properties>
>>
>>   <dependencies>
>>     <dependency>
>>       <groupId>junit</groupId>
>>       <artifactId>junit</artifactId>
>>       <version>3.8.1</version>
>>       <scope>test</scope>
>>     </dependency>
>>     <dependency>
>>       <groupId>org.apache.mahout</groupId>
>>       <artifactId>mahout-core</artifactId>
>>       <version>0.8</version>
>>       <type>jar</type>
>>       <exclusions>
>>         <exclusion>
>>           <artifactId>hadoop-core</artifactId>
>>           <groupId>org.apache.hadoop</groupId>
>>         </exclusion>
>>       </exclusions>
>>     </dependency>
>>     <dependency>
>>       <groupId>com.google.code.gson</groupId>
>>       <artifactId>gson</artifactId>
>>       <version>2.2.4</version>
>>       <type>jar</type>
>>     </dependency>
>>     <dependency>
>>       <groupId>org.apache.mahout</groupId>
>>       <artifactId>mahout-integration</artifactId>
>>       <version>0.8</version>
>>       <type>jar</type>
>>       <exclusions>
>>         <exclusion>
>>           <artifactId>hadoop-core</artifactId>
>>           <groupId>org.apache.hadoop</groupId>
>>         </exclusion>
>>       </exclusions>
>>     </dependency>
>>     <dependency>
>>         <groupId>org.apache.hadoop</groupId>
>>         <artifactId>hadoop-client</artifactId>
>>         <version>2.2.0</version>
>>     </dependency>
>>   </dependencies>
>> </project>
>>
>> How/where do I check to see whether my hadoop cluster is 2.2.0? Sorry, I
>> am new at this.
>>
>>
>>
>>
>> On Wednesday, December 11, 2013 11:56 AM, Gokhan Capan <gkhncpn@gmail.com>
>> wrote:
>>
>> Could you check the following?
>>
>> Are you sure that your hadoop cluster is hadoop 2.2.0?
>> Are you sure other dependencies of your project do not have a transitive
>> dependency to hadoop?
>>
>> Gokhan
>>
>>
>>
>> On Wed, Dec 11, 2013 at 9:46 PM, Hi There <srudamaster@yahoo.com> wrote:
>>
>> > I tried to run SparseVectorsFromSequenceFiles, specifying a directory
>> with
>> > sequence files, and I got the following error:
>> >
>> > java.lang.Exception: java.lang.IncompatibleClassChangeError: Found
>> > interface org.apache.hadoop.mapreduce.Counter, but class was expected
>> >
>> > Here is a relevant snippet of my pom file for maven:
>> >
>> > <dependency>
>> >       <groupId>org.apache.mahout</groupId>
>> >       <artifactId>mahout-core</artifactId>
>> >       <version>0.8</version>
>> >       <type>jar</type>
>> >       <exclusions>
>> >         <exclusion>
>> >           <artifactId>hadoop-core</artifactId>
>> >           <groupId>org.apache.hadoop</groupId>
>> >         </exclusion>
>> >       </exclusions>
>> > </dependency>
>> > <dependency>
>> >       <groupId>org.apache.mahout</groupId>
>> >       <artifactId>mahout-integration</artifactId>
>> >       <version>0.8</version>
>> >       <type>jar</type>
>> >       <exclusions>
>> >         <exclusion>
>> >           <artifactId>hadoop-core</artifactId>
>> >           <groupId>org.apache.hadoop</groupId>
>> >         </exclusion>
>> >       </exclusions>
>> > </dependency>
>> > <dependency>
>> >         <groupId>org.apache.hadoop</groupId>
>> >         <artifactId>hadoop-client</artifactId>
>> >         <version>2.2.0</version>
>> > </dependency>
>> >
>> >
>> > How do I change this to use the correct version(s) of mahout/hadoop?
>> >
>> > Thanks!
>> >
>> >
>> >
>> >
>> > On Tuesday, December 10, 2013 1:03 PM, Gokhan Capan <gkhncpn@gmail.com>
>> > wrote:
>> >
>> > I meant that you shouldn't need to modify mahout's dependencies, just
>> mvn
>> > package and it should work against hadoop 2.2.0 (Yeah, 2.2.0 is not
>> alpha)
>> >
>> > Quoting from
>> >
>> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
>> > "First, we ensure binary compatibility to the applications that use old
>> > mapred APIs. This means that applications which were built against MRv1
>> > mapred APIs can run directly on YARN without recompilation, merely by
>> > pointing them to an Apache Hadoop 2.x cluster via configuration."
>> >
>> > If you encounter with any problems, just let the list know.
>> >
>> > Best
>> >
>> >
>> >
>> > > On Dec 9, 2013, at 9:40 PM, Hi There <srudamaster@yahoo.com> wrote:
>> > >
>> > > Hi Gokhan,
>> > >
>> > > My project currently fetches every dependency through Maven--is there
>> > any way I can grab the version you mentioned that way?
>> > >
>> > > In that vein, I am using the following version of hadoop:
>> > > <dependency>
>> > >        <groupId>org.apache.hadoop</groupId>
>> > >        <artifactId>hadoop-client</artifactId>
>> > >        <version>2.2.0</version>
>> > > </dependency>
>> > >
>> > >
>> > > That's not alpha, right?
>> > >
>> > > Thanks!
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Monday, December 9, 2013 10:05 AM, Gokhan Capan <gkhncpn@gmail.com
>> >
>> > wrote:
>> > >
>> > > Mahout actually should work with hadoop-2 stable without recompiling,
>> > > not with hadoop-2 alpha though.
>> > >
>> > > We're, by the way, currently in the process of adding support to build
>> > > mahout with hadoop-2.
>> > >
>> > > Please see mahout-1354 for the relevant issue
>> > >
>> > > Sent from my iPhone
>> > >
>> > >
>> > >> On Dec 9, 2013, at 19:54, Hi There <srudamaster@yahoo.com> wrote:
>> > >>
>> > >> Is Dec 2013 still the intended release date of the next mahout
>> release
>> > that will be compatible with Hadoop 2.2.0?
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Thursday, November 21, 2013 12:36 PM, Suneel Marthi <
>> > suneel_marthi@yahoo.com> wrote:
>> > >>
>> > >> Targeted for Dec 2013.
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Thursday, November 21, 2013 3:26 PM, Hi There <
>> srudamaster@yahoo.com>
>> > wrote:
>> > >>
>> > >> Thanks for the reply! Is there a timeline for then the next release
>> > will be?
>> > >>
>> > >>
>> > >> Thanks,
>> > >> Victor
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Tuesday, November 19, 2013 7:30 PM, Suneel Marthi <
>> > suneel_marthi@yahoo.com> wrote:
>> > >>
>> > >> Hi Victor,
>> > >>
>> > >> Future releases of Mahout will support Hadoop 2.x, the present
>> codebase
>> > still only supports Hadoop 1.x.
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Tuesday, November 19, 2013 1:42 PM, Hi There <
>> srudamaster@yahoo.com>
>> > wrote:
>> > >>
>> > >>
>> > >>
>> > >> Hello,
>> > >>
>> > >> I recently upgraded to hadoop's
>> > >> newest release, and it seems one of their interfaces has changed, and
>> > >> when I try to create sparse vectors from sequence files, I get the
>> > >> following exception:
>> > >>
>> > >> java.lang.IncompatibleClassChangeError: Found interface
>> > org.apache.hadoop.mapreduce.Counter, but class was expected
>> > >>
>> > >> I can include more of the stack trace if necessary.
>> > >>
>> > >> Are there any plans in the immediate future to upgrade mahout to be
>> > compatible with the newest hadoop release?
>> > >>
>> > >> Thanks,
>> > >> Victor
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message