[ https://issues.apache.org/jira/browse/CRUNCH-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210610#comment-15210610
]
Josh Wills commented on CRUNCH-598:
-----------------------------------
[~desmit] what do you imagine as a fix here? A constructor argument for DefaultJoinStrategy
(and possibly ShardedJoinStrategy?)
> scaleFactor for JoinStrategy
> ----------------------------
>
> Key: CRUNCH-598
> URL: https://issues.apache.org/jira/browse/CRUNCH-598
> Project: Crunch
> Issue Type: Improvement
> Reporter: Stefan De Smit
> Priority: Minor
>
> the scaleFactor method has a big influence on planner.
> For joins, there currently isn't a clean way to set this, while it often is required,
as a join can have a big multiply factor.
> for the DefaultJoinStrategy, it's possible to add a custom JoinFn with proper scaleFactor,
or just extend the default InnerJoinFn with a scaleFactor.
> For the ShardedJoinStrategy, this isn't possible, while it often is needed more (as ShardedJoin
is especially handy for 1 to really many).
> For the default ConstantShardingStrategy, it might make sense to use the numShards also
as scalingFactor for left side. as that's kind of what happens: emit every left entry numShards
times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
|