mrunit-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alex bohr <alexjb...@gmail.com>
Subject Re: Trouble testing custom sortComparator
Date Tue, 14 Jan 2014 17:29:47 GMT
Thanks guys!
I did end up writing my own serializer (wasn't as complicated as I thought)
but Tom White's code, of course, is cleaner.  That example will help.

So can anyone explain a use case for the method "setKeyOrderComparator" in
the mapReduceDriver class?
http://mrunit.apache.org/documentation/javadocs/1.0.0/org/apache/hadoop/mrunit/MapReduceDriverBase.html#setKeyOrderComparator(org.apache.hadoop.io.RawComparator)
I'm not understanding when/why I would use that method in a test case.

Thanks again for the advice, much appreciated!

ps.  Totally different topic I did run some tests yesterday of the full job
on our cluster with the raw comparator vs the old full-object comparator
and the old comparator ran slightly faster...?!?  I'm going to blame it on
the cluster being very busy at the time of testing.


On Tue, Jan 14, 2014 at 7:19 AM, Tom Wheeler <twheeler@cloudera.com> wrote:

> On Tue, Jan 14, 2014 at 3:51 AM, Bertrand Dechoux <dechouxb@gmail.com>
> wrote:
> > I would also tend to say that the best would be to test the comparator
> > separately.
>
> It's pretty straightforward to do this and is also the approach that
> Tom White took in Hadoop:The Definitive Guide. Here's the relevant
> example from his GitHub repo:
>
>
> https://github.com/tomwhite/hadoop-book/blob/master/ch04/src/test/java/TextPairTest.java
>
> The utility methods he uses to convert a Writable instance to/from an
> array of bytes are straightforward::
>
>
> https://github.com/tomwhite/hadoop-book/blob/master/ch04/src/test/java/WritableTestBase.java
>
> Hope that helps,
>
> Tom Wheeler
>

Mime
View raw message