[ https://issues.apache.org/jira/browse/BEAM-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903135#comment-15903135
]
Eugene Kirpichov commented on BEAM-68:
--------------------------------------
I think this needs a discussion on the Beam dev@ mailing list. We should have a general approach
to annotating pipeline elements with runner-specific information. I don't think "annotate
the pipeline and let runners ignore it" is a good approach; the main reason being that this
would violate the abstraction boundary, where a pipeline is first constructed in a runner-agnostic
way (in fact while the runner is not even available), and then run. E.g. the set of all possible
runner-specific annotations is not known in advance: while "step parallelism limit" seems
relatively generic, suppose if say Apex allowed you to set an "Apex frobnication level" parameter
on a step - it would look pretty weird if the Beam pipeline API had a withApexFrobnicationLevel
method on a ParDo, and would introduce an illegal dependency from Beam to Apex.
> Support for limiting parallelism of a step
> ------------------------------------------
>
> Key: BEAM-68
> URL: https://issues.apache.org/jira/browse/BEAM-68
> Project: Beam
> Issue Type: New Feature
> Components: beam-model
> Reporter: Daniel Halperin
>
> Users may want to limit the parallelism of a step. Two classic uses cases are:
> - User wants to produce at most k files, so sets TextIO.Write.withNumShards(k).
> - External API only supports k QPS, so user sets a limit of k/(expected QPS/step) on
the ParDo that makes the API call.
> Unfortunately, there is no way to do this effectively within the Beam model. A GroupByKey
with exactly k keys will guarantee that only k elements are produced, but runners are free
to break fusion in ways that each element may be processed in parallel later.
> To implement this functionaltiy, I believe we need to add this support to the Beam Model.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
|