mrunit-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Wheeler <twhee...@cloudera.com>
Subject Re: How to set number of reduce tasks in MRUnit's mocked context object
Date Thu, 29 Nov 2012 03:10:13 GMT
Hi Dipesh,

OK, I think I understand what you're saying. I am going to restate it
just so you'll be sure I've got it.

Your mapper (or reducer) is trying to check the return value of the
context.getNumReduceTasks() method, but it's returning 0 in all cases.
 Although this wouldn't be an issue for most unit tests, your mapper
is doing some computation on this value so you need MRUnit to return
something other than 0 so you can test your code.  Does that sound
right?

If so, I cannot say offhand whether what you're seeing is a bug or a
feature that just hasn't been implemented yet.  I think I can offer a
workaround for you try, though it may be kind of a hack.

Whenever you call methods like setNumReduceTasks, it's really just a
convenient way of setting a property that Hadoop interprets.
According to the Hadoop Streaming guide [1], the corresponding
property here ought to be num.reduce.tasks.  Therefore, instead of
checking for getNumReduceTasks() in your mapper code, try checking the
return value of this:

    context.getConfiguration().get("mapreduce.job.reduces")

And then in the setup of your corresponding unit test, set that value
to whatever you want it to be:

    mapDriver.getConfiguration().setInt("mapreduce.job.reduces", 1);

I've verified that the property set this way in MRUnit 0.9.0 is
returned with the same value, though I didn't verify much beyond that.

[1] http://hadoop.apache.org/docs/mapreduce/current/streaming.html#Specifying+the+Number+of+Reducers

On Wed, Nov 28, 2012 at 8:06 PM, Dipesh Khakhkhar
<dipeshsoftware@gmail.com> wrote:
> Hi Tom,
>
> Thanks for replying. I completely agree with you - there will be only one
> Reduce task in unit test and when we query the mock object to get number of
> reduce task it should return 1 instead of zero.
>
> I'm using to calculate a custom counter and since mocked Context object
> returns it 0 my test is failing.
>
> Can we set it externally this value using MRUnit 0.9*?
>
> Thanks.
> -Dipesh

Mime
View raw message