beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xu Mingmin (JIRA)" <>
Subject [jira] [Commented] (BEAM-68) Support for limiting parallelism of a step
Date Sun, 05 Mar 2017 22:48:33 GMT


Xu Mingmin commented on BEAM-68:

Notice this task when tuning a Beam job on Flink.
Would like to bring another perspective, that users want to have more control on the parallelism
of a data pipeline, to allocate more resource for the busy steps, and less for the costless.
A fixed parallelism could have performance bottleneck, several use cases like:
1. source from a Kafka topic, the parallelism could not be larger then topic partition number;
similar for other splittable IOs?
2. fewer grouped keys than parallelism;
3. process on a small portion from large input;
4. +1 for case2, to address quota limitation on external dependencies;

> Support for limiting parallelism of a step
> ------------------------------------------
>                 Key: BEAM-68
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Daniel Halperin
> Users may want to limit the parallelism of a step. Two classic uses cases are:
> - User wants to produce at most k files, so sets TextIO.Write.withNumShards(k).
> - External API only supports k QPS, so user sets a limit of k/(expected QPS/step) on
the ParDo that makes the API call.
> Unfortunately, there is no way to do this effectively within the Beam model. A GroupByKey
with exactly k keys will guarantee that only k elements are produced, but runners are free
to break fusion in ways that each element may be processed in parallel later.
> To implement this functionaltiy, I believe we need to add this support to the Beam Model.

This message was sent by Atlassian JIRA

View raw message