mrunit-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Trouble testing custom sortComparator
Date Wed, 15 Jan 2014 16:00:52 GMT
Can you send me a full code example that reproduces the error?

I believe you are using KeyOrderComparator correctly.

Brock


On Tue, Jan 14, 2014 at 11:29 AM, alex bohr <alexjbohr@gmail.com> wrote:

> Thanks guys!
> I did end up writing my own serializer (wasn't as complicated as I
> thought) but Tom White's code, of course, is cleaner.  That example will
> help.
>
> So can anyone explain a use case for the method "setKeyOrderComparator" in
> the mapReduceDriver class?
>
> http://mrunit.apache.org/documentation/javadocs/1.0.0/org/apache/hadoop/mrunit/MapReduceDriverBase.html#setKeyOrderComparator(org.apache.hadoop.io.RawComparator)
> I'm not understanding when/why I would use that method in a test case.
>
> Thanks again for the advice, much appreciated!
>
> ps.  Totally different topic I did run some tests yesterday of the full
> job on our cluster with the raw comparator vs the old full-object
> comparator and the old comparator ran slightly faster...?!?  I'm going to
> blame it on the cluster being very busy at the time of testing.
>
>
> On Tue, Jan 14, 2014 at 7:19 AM, Tom Wheeler <twheeler@cloudera.com>wrote:
>
>> On Tue, Jan 14, 2014 at 3:51 AM, Bertrand Dechoux <dechouxb@gmail.com>
>> wrote:
>> > I would also tend to say that the best would be to test the comparator
>> > separately.
>>
>> It's pretty straightforward to do this and is also the approach that
>> Tom White took in Hadoop:The Definitive Guide. Here's the relevant
>> example from his GitHub repo:
>>
>>
>> https://github.com/tomwhite/hadoop-book/blob/master/ch04/src/test/java/TextPairTest.java
>>
>> The utility methods he uses to convert a Writable instance to/from an
>> array of bytes are straightforward::
>>
>>
>> https://github.com/tomwhite/hadoop-book/blob/master/ch04/src/test/java/WritableTestBase.java
>>
>> Hope that helps,
>>
>> Tom Wheeler
>>
>
>


-- 
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org

Mime
View raw message