mrunit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Beech (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MRUNIT-114) should support calling configure on sorting and grouping comparators if applicable
Date Wed, 13 Jun 2012 15:33:25 GMT

    [ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294522#comment-13294522
] 

Dave Beech commented on MRUNIT-114:
-----------------------------------

Hi Jim - I'm working with the KeyFieldBasedComparator at the moment and I've found a couple
more issues you might need to consider. 

1. Because the shuffle method in MapReduceDriverBase calls the key ordering comparator using
compare(Object,Object), it does not call the correct method on classes which only override
and do all the work in compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2), like
KeyFieldBasedComparator. 

It might be better to check whether the two keys are BinaryComparable, casting, and calling
this method if possible. Even if a user's comparator doesn't override it, the default implementation
of this in WritableComparator would still be OK. 

2. KeyFieldBasedComparator doesn't get configured properly even after the change in 1344239,
because ReflectionUtils.newInstance requires the provided config to be a JobConf in order
for it to call the configure method on a JobConfigurable object. Of course, the trouble is
that simply wrapping getConfiguration as new JobConf(getConfiguration()) isn't a great thing
to do as MapReduceDriverBase is shared by the mapreduce api version... 





                
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
>                 Key: MRUNIT-114
>                 URL: https://issues.apache.org/jira/browse/MRUNIT-114
>             Project: MRUnit
>          Issue Type: Bug
>            Reporter: Jacob Metcalf
>            Assignee: Jim Donofrio
>             Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the
configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
>             // Configure grouping and sorting comparators
>    if (keyGroupComparator instanceof Configured ) {
>        ((Configured)keyGroupComparator).setConf( configuration );
>    }
>    if (keyValueOrderComparator instanceof Configured) {
>        ((Configured)keyValueOrderComparator).setConf( configuration );
>    }
> To get the config applied even though they are Configured objects.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message