beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-68) Support for limiting parallelism of a step
Date Mon, 06 Mar 2017 12:05:33 GMT

    [ https://issues.apache.org/jira/browse/BEAM-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897173#comment-15897173
] 

Eugene Kirpichov commented on BEAM-68:
--------------------------------------

This feels more runner-specific to me, because e.g. Dataflow runner is able to dynamically
choose parallelism and allocate more or fewer resources depending on how much parallelism
is needed for the currently running steps (see https://cloud.google.com/blog/big-data/2016/03/comparing-cloud-dataflow-autoscaling-to-spark-and-hadoop).
I don't know what controls Flink in particular provides for this, though.

> Support for limiting parallelism of a step
> ------------------------------------------
>
>                 Key: BEAM-68
>                 URL: https://issues.apache.org/jira/browse/BEAM-68
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Daniel Halperin
>
> Users may want to limit the parallelism of a step. Two classic uses cases are:
> - User wants to produce at most k files, so sets TextIO.Write.withNumShards(k).
> - External API only supports k QPS, so user sets a limit of k/(expected QPS/step) on
the ParDo that makes the API call.
> Unfortunately, there is no way to do this effectively within the Beam model. A GroupByKey
with exactly k keys will guarantee that only k elements are produced, but runners are free
to break fusion in ways that each element may be processed in parallel later.
> To implement this functionaltiy, I believe we need to add this support to the Beam Model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message