mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhang, Pengchu" <pzh...@sandia.gov>
Subject RE: [EXTERNAL] Re: Mapreduce job failed
Date Tue, 25 Feb 2014 20:37:50 GMT
Built and run successfully on clustering (kmeans) with svn trunk and hadoop2.2.0. Thanks for
you nice work. -Pengchu

From: Gokhan Capan [mailto:gkhncpn@gmail.com]
Sent: Friday, February 21, 2014 12:21 PM
To: Zhang, Pengchu
Subject: Re: [EXTERNAL] Re: Mapreduce job failed

Great then, I'm gonna commit it on Monday

Sent from my iPhone

On Feb 21, 2014, at 17:46, "Zhang, Pengchu" <pzhang@sandia.gov<mailto:pzhang@sandia.gov>>
wrote:
I saw that you have tested on clustering… successfully.

That is what I want to do.

From: Gokhan Capan [mailto:gkhncpn@gmail.com]
Sent: Friday, February 21, 2014 8:37 AM
To: Zhang, Pengchu
Subject: Re: [EXTERNAL] Re: Mapreduce job failed

By the way, you can see the progress in

https://issues.apache.org/jira/browse/MAHOUT-1329


Gokhan

On Fri, Feb 21, 2014 at 5:32 PM, Zhang, Pengchu <pzhang@sandia.gov<mailto:pzhang@sandia.gov>>
wrote:
Thanks, I am looking forward to trying it. Penghcu

From: Gokhan Capan [mailto:gkhncpn@gmail.com<mailto:gkhncpn@gmail.com>]
Sent: Friday, February 21, 2014 8:30 AM

To: Zhang, Pengchu
Subject: Re: [EXTERNAL] Re: Mapreduce job failed

Hi, I tried it and it works.

I'll commit this in two days, then you can build mahout as usual and work with it without
a problem.

Best

Gokhan

On Fri, Feb 21, 2014 at 4:56 PM, Zhang, Pengchu <pzhang@sandia.gov<mailto:pzhang@sandia.gov>>
wrote:
Eclipse on Linux Ubuntu. But I can build with command line. I have installed and used Maven.
Thanks.

Pengchu

From: Gokhan Capan [mailto:gkhncpn@gmail.com<mailto:gkhncpn@gmail.com>]
Sent: Thursday, February 20, 2014 11:28 PM
To: Zhang, Pengchu

Subject: Re: [EXTERNAL] Re: Mapreduce job failed

Sure, what IDE are you using for Java development, IntelliJ?

Sent from my iPhone

On Feb 21, 2014, at 0:47, "Zhang, Pengchu" <pzhang@sandia.gov<mailto:pzhang@sandia.gov>>
wrote:
Gokhan:

It is embraced (so not posted to the user group ☺)to ask the procedure to build the patch
with Mahout and Hadoop, I have never done building some package with a patch. Thanks.


1.       Download the mahout 0.9 source codes

2.       Replace core/pom.xml with the patch 1329-3

3.       Build it with Maven?

Pengchu


From: Gokhan Capan [mailto:gkhncpn@gmail.com]
Sent: Thursday, February 20, 2014 2:40 PM
To: user@mahout.apache.org<mailto:user@mahout.apache.org>; Zhang, Pengchu
Subject: Re: [EXTERNAL] Re: Mapreduce job failed

If you have a chance to build mahout from source, could you try if that works when the patch
in MAHOUT-1329 applied? mvn packaging mahout with "-DskipTests=true" is pretty fast

Gokhan

On Thu, Feb 20, 2014 at 11:34 PM, Suneel Marthi <suneel_marthi@yahoo.com<mailto:suneel_marthi@yahoo.com>>
wrote:





On Thursday, February 20, 2014 4:26 PM, "Zhang, Pengchu" <pzhang@sandia.gov<mailto:pzhang@sandia.gov>>
wrote:

Thanks, it has been executed successfully.  Two more questions related to this:

1. This means that I have to execute Mahout for further analysis with the non-MR mode?
No, that's not the case. There have been API changes between Hadoop 2.x and Hadoop 1.x.
Mahout is not certified for Hadoop 2.x.
But most of Mahout's jobs work on Hadoop 2.x, seqdirectory could be one job that's failing
due to API incompatibility between Hadoop 1.x and Hadoop 2.x


2. It is too bad that Hadoop2.2. does not support for newer versions of Mahout. Are you aware
of that Hadoop 1.x working with Mahout 0.8 0r 0.9 on MR? I do have a large dataset to be clustered.
As mentioned earlier, Mahout 0.8/0.9 are certified for Hadoop 1.x so u shouldn't be seeing
any issues with that.

Could u open a JIRA for this issue so that its trackable?  As we r
 now working towards Mahout 1.0 and Hadoop 2.x compatibility its good that u have reported
this issue. Thanks.



Thanks.

Pengchu


-----Original Message-----
From: Suneel Marthi [mailto:suneel_marthi@yahoo.com<mailto:suneel_marthi@yahoo.com>]
Sent: Thursday, February 20, 2014 1:17 PM
To: user@mahout.apache.org<mailto:user@mahout.apache.org>
Subject: [EXTERNAL] Re: Mapreduce job failed

... and the reason for this failing is that 'TaskAttemptContext' which was a Class in Hadoop
1.x has now become an interface in Hadoop 2.2.

Suggest that u execute this job in non-MR mode with '-xm
 sequential'.




On Thursday, February 20, 2014 2:26 PM, Suneel Marthi <suneel_marthi@yahoo.com<mailto:suneel_marthi@yahoo.com>>
wrote:

Seems like u r running this on HAdoop 2.2 (officially not supported for Mahout 0.8 or 0.9),
work around is to run this in sequential mode with "-xm sequential".







On Thursday, February 20, 2014 1:36 PM, "Zhang, Pengchu" <pzhang@sandia.gov<mailto:pzhang@sandia.gov>>
wrote:

Hello, I am trying to "seqdirirectory" with mahout (0.8 and 0.9) on my Linux box with Hadoop
(2.2.0) but keeping failed
 consistently.

I tested Hadoop with the Hadoop example pi and wordcount, both worked well.

With a simple text file or directory with multiple text files, e.g., Shakespeare_text, I got
the same message of failure.

>mahout seqdirectory --input /Shakespeare_txet --output
>/Shakespeare-seqdir --charset utf-8

$ mahout seqdirectory --input /shakespeare_text --output /shakespeare-seqdir --charset utf-8
MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
Running on hadoop, using /home/pzhang/hadoop-2.2.0/bin/hadoop and HADOOP_CONF_DIR=/home/pzhang/hadoop-2.2.0/etc/hadoop
MAHOUT-JOB: /home/pzhang/MAHOUT_HOME/mahout-examples-0.8-job.jar
14/02/20 11:29:42 INFO common.AbstractJob: Command line arguments: {--charset=[utf-8], --chunkSize=[64],
 --endPhase=[2147483647<tel:%5B2147483647>], --fileFilterClass=[org.apache.mahout.text.PrefixAdditionFilter],
--input=[/shakespeare_text], --keyPrefix=[], --method=[mapreduce], --output=[/shakespeare-seqdir],
--startPhase=[0], --tempDir=[temp]}
14/02/20 11:29:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
14/02/20 11:29:42 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead,
use mapreduce.input.fileinputformat.inputdir
14/02/20 11:29:42 INFO Configuration.deprecation: mapred.compress.map.output is deprecated.
Instead, use mapreduce.map.output.compress
14/02/20 11:29:42 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead,
use mapreduce.output.fileoutputformat.outputdir
14/02/20 11:29:43 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032<http://0.0.0.0:8032>
14/02/20 11:29:44 INFO input.FileInputFormat: Total input paths to process : 43
14/02/20 11:29:44 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with
: CompletedNodes: 1, size left: 5284832
14/02/20 11:29:44 INFO mapreduce.JobSubmitter: number of splits:1
14/02/20 11:29:44 INFO Configuration.deprecation: user.name<http://user.name> is deprecated.
Instead, use mapreduce.job.user.name<http://mapreduce.job.user.name>
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.output.compress is deprecated. Instead,
use mapreduce.output.fileoutputformat.compress
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead,
use mapreduce.job.reduces
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.output.value.class is
 deprecated. Instead, use mapreduce.job.output.value.class
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated.
Instead, use mapreduce.map.output.value.class
14/02/20 11:29:44 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead,
use mapreduce.job.map.class
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.job.name<http://mapred.job.name>
is deprecated. Instead, use mapreduce.job.name<http://mapreduce.job.name>
14/02/20 11:29:44 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated.
Instead, use mapreduce.job.inputformat.class
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead,
use mapreduce.input.fileinputformat.split.maxsize
14/02/20 11:29:44 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated.
Instead, use mapreduce.job.outputformat.class
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead,
use mapreduce.job.maps
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead,
use mapreduce.job.output.key.class
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated.
Instead, use mapreduce.map.output.key.class
14/02/20 11:29:44 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead,
use mapreduce.job.working.dir
14/02/20 11:29:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1392919123773_0004
14/02/20 11:29:45 INFO impl.YarnClientImpl: Submitted application application_1392919123773_0004
to ResourceManager at /0.0.0.0:8032<http://0.0.0.0:8032>
14/02/20 11:29:45 INFO mapreduce.Job: The url to track the job: http://savm0072lx.sandia.gov:8088/proxy/application_1392919123773_0004/
14/02/20 11:29:45 INFO mapreduce.Job: Running job: job_1392919123773_0004
14/02/20 11:29:53 INFO mapreduce.Job: Job job_1392919123773_0004 running in uber mode : false
14/02/20 11:29:53 INFO mapreduce.Job:  map 0% reduce 0%
14/02/20 11:29:58 INFO mapreduce.Job: Task Id : attempt_1392919123773_0004_m_000000_0, Status
: FAILED
Error: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.java:164)
    at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.<init>(CombineFileRecordReader.java:126)
    at org.apache.mahout.text.MultipleTextFileInputFormat.createRecordReader(MultipleTextFileInputFormat.java:43)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:491)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:416)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at
 org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:534)
    at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.java:155)
    ... 10 more
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext,
but class was
 expected
    at org.apache.mahout.text.WholeFileRecordReader.<init>(WholeFileRecordReader.java:52)
    ... 15 more

Any suggestion is helpful.

Thanks.

Pengchu



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message