hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Dusso <pmdu...@gmail.com>
Subject Problems setting custom class for the Pluggable Sort in MapReduce Next Generation
Date Tue, 20 May 2014 12:27:06 GMT

I'm developing a custom map output buffer which uses replacement selection
instead of quicksort. It's available
It is based on the new pluggable interface from the JIRA number

I've been testing it in a single-node installation with success. I
configure the job during its creation like this:

*  conf.set("io.serializations",
* conf.set("mapreduce.job.map.output.collector.class",

I used to generate a runnable jar and run it normally as java -jar ...  But
now I would like to try it in a multinode cluster (which is working with
normal jobs). I remove this hardcoded configuration and start calling the
jar like:

*hadoop jar jars/wordCount.jar
/wordcount/words /wordcount/output/out*

But I can't get this to work. I keep getting a ClassNotFoundException:

Error: java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: Class
pluggable.MapOutputHeapWithMetadataHeap.class not found
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1927)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:383)
 at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class pluggable.MapOutputHeapWithMetadataHeap.class not found
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1919)
 ... 10 more
Caused by: java.lang.ClassNotFoundException: Class
pluggable.MapOutputHeapWithMetadataHeap.class not found
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 11 more

I have two projects: one for jobs like wordCount, grep, etc. and one where
I'm developing my custom output buffer (the one in the bitbucket linked
above). Because of this, I tried different jar configurations:

   - Project jobs having a *project* dependency in Eclipse. Export runnable
   jar with packaged required libraries and also copied as a folder
   - Project jobs adds a jar generate from custom output buffer project
   - Fat jar generated with mvn in project jobs.

All of those failed. I would appreciate any help, since it seems to have
very few information about this online. If I'm missing some important
information, please let know I will bring it.

Best regards,

Pedro Martins Dusso

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message