[ https://issues.apache.org/jira/browse/CRUNCH-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584375#comment-13584375
]
Gabriel Reid commented on CRUNCH-167:
-------------------------------------
Sounds very cool -- I'm going to try to take a closer look at this this weekend. It would
be great if we could take advantage of this to get CRUNCH-51 taken care of
> Sort.sortTuples and related methods write out duplicate values
> --------------------------------------------------------------
>
> Key: CRUNCH-167
> URL: https://issues.apache.org/jira/browse/CRUNCH-167
> Project: Crunch
> Issue Type: Bug
> Affects Versions: 0.4.0, 0.5.0
> Reporter: Josh Wills
> Fix For: 0.6.0
>
> Attachments: CRUNCH-167.patch
>
>
> I noticed when I was debugging CRUNCH-166 that the strategy that the Sort.sortPairs,
sortTrips, etc. methods are using has the potential to write out duplicate values in cases
where we are only sorting/grouping on a subset of the fields, because all of the records that
have the same value for those sub-fields will be called as part of the same reduce() call,
where only a single one of the records that had the same set of values for those sub-fields
will be used as the key, and the rest of the values will have been thrown away.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
|