mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hi There <srudamas...@yahoo.com>
Subject Re: Mahout and Hadoop 2.2.0
Date Wed, 11 Dec 2013 20:03:41 GMT
Here are the full contents of my pom file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>clustertest</groupId>
  <artifactId>clustertest</artifactId>
  <version>1.0</version>
  <packaging>jar</packaging>

  <name>clustertest</name>
  <url>http://maven.apache.org</url>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.mahout</groupId>
      <artifactId>mahout-core</artifactId>
      <version>0.8</version>
      <type>jar</type>
      <exclusions>
        <exclusion>
          <artifactId>hadoop-core</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>com.google.code.gson</groupId>
      <artifactId>gson</artifactId>
      <version>2.2.4</version>
      <type>jar</type>
    </dependency>
    <dependency>
      <groupId>org.apache.mahout</groupId>
      <artifactId>mahout-integration</artifactId>
      <version>0.8</version>
      <type>jar</type>
      <exclusions>
        <exclusion>
          <artifactId>hadoop-core</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.2.0</version>
    </dependency>
  </dependencies>
</project>

How/where do I check to see whether my hadoop cluster is 2.2.0? Sorry, I am new at this.




On Wednesday, December 11, 2013 11:56 AM, Gokhan Capan <gkhncpn@gmail.com> wrote:
 
Could you check the following?

Are you sure that your hadoop cluster is hadoop 2.2.0?
Are you sure other dependencies of your project do not have a transitive
dependency to hadoop?

Gokhan



On Wed, Dec 11, 2013 at 9:46 PM, Hi There <srudamaster@yahoo.com> wrote:

> I tried to run SparseVectorsFromSequenceFiles, specifying a directory with
> sequence files, and I got the following error:
>
> java.lang.Exception: java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.Counter, but class was expected
>
> Here is a relevant snippet of my pom file for maven:
>
> <dependency>
>       <groupId>org.apache.mahout</groupId>
>       <artifactId>mahout-core</artifactId>
>       <version>0.8</version>
>       <type>jar</type>
>       <exclusions>
>         <exclusion>
>           <artifactId>hadoop-core</artifactId>
>           <groupId>org.apache.hadoop</groupId>
>         </exclusion>
>       </exclusions>
> </dependency>
> <dependency>
>       <groupId>org.apache.mahout</groupId>
>       <artifactId>mahout-integration</artifactId>
>       <version>0.8</version>
>       <type>jar</type>
>       <exclusions>
>         <exclusion>
>           <artifactId>hadoop-core</artifactId>
>           <groupId>org.apache.hadoop</groupId>
>         </exclusion>
>       </exclusions>
> </dependency>
> <dependency>
>         <groupId>org.apache.hadoop</groupId>
>         <artifactId>hadoop-client</artifactId>
>         <version>2.2.0</version>
> </dependency>
>
>
> How do I change this to use the correct version(s) of mahout/hadoop?
>
> Thanks!
>
>
>
>
> On Tuesday, December 10, 2013 1:03 PM, Gokhan Capan <gkhncpn@gmail.com>
> wrote:
>
> I meant that you shouldn't need to modify mahout's dependencies, just mvn
> package and it should work against hadoop 2.2.0 (Yeah, 2.2.0 is not alpha)
>
> Quoting from
> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> "First, we ensure binary compatibility to the applications that use old
> mapred APIs. This means that applications which were built against MRv1
> mapred APIs can run directly on YARN without recompilation, merely by
> pointing them to an Apache Hadoop 2.x cluster via configuration."
>
> If you encounter with any problems, just let the list know.
>
> Best
>
>
>
> > On Dec 9, 2013, at 9:40 PM, Hi There <srudamaster@yahoo.com> wrote:
> >
> > Hi Gokhan,
> >
> > My project currently fetches every dependency through Maven--is there
> any way I can grab the version you mentioned that way?
> >
> > In that vein, I am using the following version of hadoop:
> > <dependency>
> >        <groupId>org.apache.hadoop</groupId>
> >        <artifactId>hadoop-client</artifactId>
> >        <version>2.2.0</version>
> > </dependency>
> >
> >
> > That's not alpha, right?
> >
> > Thanks!
> >
> >
> >
> >
> >
> > On Monday, December 9, 2013 10:05 AM, Gokhan Capan <gkhncpn@gmail.com>
> wrote:
> >
> > Mahout actually should work with hadoop-2 stable without recompiling,
> > not with hadoop-2 alpha though.
> >
> > We're, by the way, currently in the process of adding support to build
> > mahout with hadoop-2.
> >
> > Please see mahout-1354 for the relevant issue
> >
> > Sent from my iPhone
> >
> >
> >> On Dec 9, 2013, at 19:54, Hi There <srudamaster@yahoo.com> wrote:
> >>
> >> Is Dec 2013 still the intended release date of the next mahout release
> that will be compatible with Hadoop 2.2.0?
> >>
> >>
> >>
> >>
> >> On Thursday, November 21, 2013 12:36 PM, Suneel Marthi <
> suneel_marthi@yahoo.com> wrote:
> >>
> >> Targeted for Dec 2013.
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thursday, November 21, 2013 3:26 PM, Hi There <srudamaster@yahoo.com>
> wrote:
> >>
> >> Thanks for the reply! Is there a timeline for then the next release
> will be?
> >>
> >>
> >> Thanks,
> >> Victor
> >>
> >>
> >>
> >>
> >> On Tuesday, November 19, 2013 7:30 PM, Suneel Marthi <
> suneel_marthi@yahoo.com> wrote:
> >>
> >> Hi Victor,
> >>
> >> Future releases of Mahout will support Hadoop 2.x, the present codebase
> still only supports Hadoop 1.x.
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Tuesday, November 19, 2013 1:42 PM, Hi There <srudamaster@yahoo.com>
> wrote:
> >>
> >>
> >>
> >> Hello,
> >>
> >> I recently upgraded to hadoop's
> >> newest release, and it seems one of their interfaces has changed, and
> >> when I try to create sparse vectors from sequence files, I get the
> >> following exception:
> >>
> >> java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.Counter, but class was expected
> >>
> >> I can include more of the stack trace if necessary.
> >>
> >> Are there any plans in the immediate future to upgrade mahout to be
> compatible with the newest hadoop release?
> >>
> >> Thanks,
> >> Victor
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message