mrunit-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Beech <d...@paraliatech.com>
Subject Re: How to set number of reduce tasks in MRUnit's mocked context object
Date Thu, 29 Nov 2012 12:16:42 GMT
Hi Dipesh

The Context in a mrunit test is actually a mock object (created with
Mockito). Only some of the methods are set-up internally to provide return
values, and getNumReduceTasks isn't one of them. But, you can set this up
yourself in test code.

e.g.
Mockito.when(mapDriver.getContext().getNumReduceTasks()).thenReturn(10);

Cheers,
Dave


On 29 November 2012 03:10, Tom Wheeler <twheeler@cloudera.com> wrote:

> Hi Dipesh,
>
> OK, I think I understand what you're saying. I am going to restate it
> just so you'll be sure I've got it.
>
> Your mapper (or reducer) is trying to check the return value of the
> context.getNumReduceTasks() method, but it's returning 0 in all cases.
>  Although this wouldn't be an issue for most unit tests, your mapper
> is doing some computation on this value so you need MRUnit to return
> something other than 0 so you can test your code.  Does that sound
> right?
>
> If so, I cannot say offhand whether what you're seeing is a bug or a
> feature that just hasn't been implemented yet.  I think I can offer a
> workaround for you try, though it may be kind of a hack.
>
> Whenever you call methods like setNumReduceTasks, it's really just a
> convenient way of setting a property that Hadoop interprets.
> According to the Hadoop Streaming guide [1], the corresponding
> property here ought to be num.reduce.tasks.  Therefore, instead of
> checking for getNumReduceTasks() in your mapper code, try checking the
> return value of this:
>
>     context.getConfiguration().get("mapreduce.job.reduces")
>
> And then in the setup of your corresponding unit test, set that value
> to whatever you want it to be:
>
>     mapDriver.getConfiguration().setInt("mapreduce.job.reduces", 1);
>
> I've verified that the property set this way in MRUnit 0.9.0 is
> returned with the same value, though I didn't verify much beyond that.
>
> [1]
> http://hadoop.apache.org/docs/mapreduce/current/streaming.html#Specifying+the+Number+of+Reducers
>
> On Wed, Nov 28, 2012 at 8:06 PM, Dipesh Khakhkhar
> <dipeshsoftware@gmail.com> wrote:
> > Hi Tom,
> >
> > Thanks for replying. I completely agree with you - there will be only one
> > Reduce task in unit test and when we query the mock object to get number
> of
> > reduce task it should return 1 instead of zero.
> >
> > I'm using to calculate a custom counter and since mocked Context object
> > returns it 0 my test is failing.
> >
> > Can we set it externally this value using MRUnit 0.9*?
> >
> > Thanks.
> > -Dipesh
>

Mime
View raw message