mrunit-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Trouble testing custom sortComparator
Date Wed, 15 Jan 2014 16:02:11 GMT
Though I do agree with the others statements that the comparator should
have it's own unit tests outside of MRUnit. Using it in MRUnit is more of
an integration tests for the comparator.


On Wed, Jan 15, 2014 at 10:00 AM, Brock Noland <brock@cloudera.com> wrote:

> Can you send me a full code example that reproduces the error?
>
> I believe you are using KeyOrderComparator correctly.
>
> Brock
>
>
> On Tue, Jan 14, 2014 at 11:29 AM, alex bohr <alexjbohr@gmail.com> wrote:
>
>> Thanks guys!
>> I did end up writing my own serializer (wasn't as complicated as I
>> thought) but Tom White's code, of course, is cleaner.  That example will
>> help.
>>
>> So can anyone explain a use case for the method "setKeyOrderComparator" in
>> the mapReduceDriver class?
>>
>> http://mrunit.apache.org/documentation/javadocs/1.0.0/org/apache/hadoop/mrunit/MapReduceDriverBase.html#setKeyOrderComparator(org.apache.hadoop.io.RawComparator)
>> I'm not understanding when/why I would use that method in a test case.
>>
>> Thanks again for the advice, much appreciated!
>>
>> ps.  Totally different topic I did run some tests yesterday of the full
>> job on our cluster with the raw comparator vs the old full-object
>> comparator and the old comparator ran slightly faster...?!?  I'm going to
>> blame it on the cluster being very busy at the time of testing.
>>
>>
>> On Tue, Jan 14, 2014 at 7:19 AM, Tom Wheeler <twheeler@cloudera.com>wrote:
>>
>>> On Tue, Jan 14, 2014 at 3:51 AM, Bertrand Dechoux <dechouxb@gmail.com>
>>> wrote:
>>> > I would also tend to say that the best would be to test the comparator
>>> > separately.
>>>
>>> It's pretty straightforward to do this and is also the approach that
>>> Tom White took in Hadoop:The Definitive Guide. Here's the relevant
>>> example from his GitHub repo:
>>>
>>>
>>> https://github.com/tomwhite/hadoop-book/blob/master/ch04/src/test/java/TextPairTest.java
>>>
>>> The utility methods he uses to convert a Writable instance to/from an
>>> array of bytes are straightforward::
>>>
>>>
>>> https://github.com/tomwhite/hadoop-book/blob/master/ch04/src/test/java/WritableTestBase.java
>>>
>>> Hope that helps,
>>>
>>> Tom Wheeler
>>>
>>
>>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>



-- 
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org

Mime
View raw message