mrunit-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alex bohr <alexjb...@gmail.com>
Subject Trouble testing custom sortComparator
Date Mon, 13 Jan 2014 19:27:11 GMT
Hello,
I'm trying to use MRunit to test a custom sortComparatorClass. I believe I
should be able to do this with the "setKeyOrderComparator" method, but when
I run the mapReduceDriver it is not calling the compare method of my custom
SortComparator class.
Pretty sure I'm doing something wrong with MRunit API.

Here's my code for the unit test:

public class UnitTests {
private static transient Log log = LogFactory.getLog(UnitTests.class);

MapReduceDriver<Text, Text, Text, Text, Text, Text> mapReduceDriver;
MapDriver<Text, Text, Text, Text> mapDriver;
ReduceDriver<Text, Text, Text, Text> reduceDriver;

@Before
public void setUp() throws InterruptedException, IOException {
    mapDriver = new MapDriver<Text, Text, Text, Text>();
    mapDriver.setMapper(new TestMapper());
    reduceDriver = new ReduceDriver<Text, Text, Text, Text>();
    reduceDriver.setReducer(new TestReducer());
    mapReduceDriver = new MapReduceDriver(new TestMapper(), new
TestReducer());
    mapReduceDriver.setKeyOrderComparator(new TestSortCompartor());
}

@Test
public void testSort() throws IOException {

    Text inputKey1 = new Text("def");
    Text inputKey2 = new Text("abc");
    Text inputValue = new Text("BlahBlahBlah");
    mapReduceDriver.addInput(inputKey1, inputValue);
    mapReduceDriver.addInput(inputKey2, inputValue);

    List<Pair<Text, Text>> output = mapReduceDriver.run();
    log.info("Got output of size "+output.size()+" with first pair =
"+output.get(0).toString());
}
}

And here's my test sortComparator:

public class TestSortCompartor extends WritableComparator{
private static transient Log log =
LogFactory.getLog(TestSortCompartor.class);
public TestSortCompartor() {
    super(Text.class, true);
}

@SuppressWarnings("rawtypes")
@Override
public int compare(WritableComparable w1, WritableComparable w2) {
    log.info("calling compare with key1 = "+w1.toString()+" and key2
"+w2.toString());
    return w1.compareTo(w2) * -1;
}
}

When I run the test I get this output: INFO 2014-01-13 09:34:27,362 [main]
(com.gradientx.gxetl.testMR.UnitTests:53) - Got output of size 2 with first
pair = (abc, BlahBlahBlah)

But there's no output from the sortComparator class - and it's not ordering
the keys in reverse, so I know my comparator class is not being called.

Can anyone advise what I'm doing wrong? Is it possible to use MRunit to
test my own comparator class? Is there a better way to make a unit test for
a custom comparator class?
FWIW, The above code is just to illustrate.  I'm making rawComparators on
complex keys and am looking for a unit test to help me debug reading in raw
bytes from the dataInputBuffer.  And I figured MRunit has already
accurately mocked the dataInputBuffer, so that's my ultimate goal.

Any advice much appreciated!

FYI, here's the relevant dependencies from my Pom:

        <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>2.0.0-mr1-cdh4.4.0</version>
        <scope>provided</scope>
    </dependency>
  <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.0.0-mr1-cdh4.4.0</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.mrunit</groupId>
        <artifactId>mrunit</artifactId>
        <version>1.0.0</version>
        <classifier>hadoop2</classifier>
    </dependency>

Mime
View raw message