chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yingzhong Xu (JIRA)" <>
Subject [jira] Created: (CHUKWA-581) Demux do not support user custom reducer package name.
Date Wed, 23 Feb 2011 02:48:38 GMT
Demux  do not support user custom reducer package name.

                 Key: CHUKWA-581
             Project: Chukwa
          Issue Type: Improvement
          Components: MR Data Processors
    Affects Versions: 0.4.0
         Environment: Ubuntu 10.0.4,Hadoop 0.20
            Reporter: Yingzhong Xu
            Priority: Critical
             Fix For: 0.4.0

it only gets {reduceType}.class from this package,as following:
bq. String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;

So our custom reducer could only be placed in that package.
Thus, When we add our {demux}.jar to Hadoop classpath, since the Hadoop framework will not
be able to merge the class files from multiple jar files with the same package name, custom
reduce class can not be found.
I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+{custom}
Here,{custom} is a part of package name.Then in parser,we can write like this:
bq. Key.setReducerType("{custom}.YourReducer")
In this way, Hadoop framework could be able to find our reduce class in this partial user
defined new package.
But I think it will be better to fix chukwa to fully support custom package name for reducer.

Thanks for Eric Yang's help.

This message is automatically generated by JIRA.
For more information on JIRA, see:


View raw message