hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Rosenstrauch (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-2002) MRUnit driver classes should provide ability to set a configuration object to be passed into the mapper/reducer
Date Wed, 11 Aug 2010 02:45:16 GMT
MRUnit driver classes should provide ability to set a configuration object to be passed into
the mapper/reducer

                 Key: MAPREDUCE-2002
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2002
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: contrib/mrunit
    Affects Versions: 0.20.2
            Reporter: David Rosenstrauch
            Priority: Minor

Short description:

Enhance the org.apache.hadoop.mrunit.mapreduce.MapDriver, ReduceDriver, and MapReduceDriver
unit test driver classes to contain "setConfiguration" and "withConfiguration" methods for
passing in user-supplied org.apache.hadoop.conf.Configuration objects, and have those configuration
objects eventually get passed on to the Context objects that are passed in to the mapper/reducer
"setup" methods.  (Rather than passing in an empty Configuration object, as is being done

Long description:

The MRUnit driver classes (i.e., MapDriver, ReduceDriver, and MapReduceDriver) ought to be
enhanced to contain methods for setting a Configuration object to be used by the mapper/reducer
being tested - i.e., setConfiguration() and withConfiguration().

The only way to effectively pass parameters into a mapper or reducer is by setting properties
on a configuration object, which the mapper/reducer can then retrieve in their "setup" step,
and use to customize its operation.  As a result, specific mappers/reducers may require the
presence of specific configuration properties/parameters in order to function correctly (or
at all).  (I am currently coding such a reducer right now.)

Testing such a mapper/reducer thus requires that the unit testing framework used provide the
ability to pass in user-supplied Configuration objects to them so that they can be tested
with appropriate parameter values.  However, MRUnit currently does not provide this ability.
 (All mappers/reducers are always passed an empty configuration object.)  And there is not
even currently any (easy) way for the end-user to fix this problem by creating a simple sub-class
that supplies this functionality, as such subclasses would require a substantial reimplementation/override
of several MRUnit framework classes.

I believe this something that is not too difficult to fix in the MRUnit framework code, however,
and would greatly help the usability of MRUnit.

Although I don't have time to code this enhancement right now, if needed/preferred I could
squeeze out some time to code up a patch for this.  If that's needed, please let me know.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message