beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Groh (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-2453) The DirectRunner should exercise all parts of a CombineFn
Date Thu, 15 Jun 2017 22:18:00 GMT
Thomas Groh created BEAM-2453:
---------------------------------

             Summary: The DirectRunner should exercise all parts of a CombineFn
                 Key: BEAM-2453
                 URL: https://issues.apache.org/jira/browse/BEAM-2453
             Project: Beam
          Issue Type: Bug
          Components: runner-direct
            Reporter: Thomas Groh
            Assignee: Thomas Groh


Specifically it should:

Create some number of accumulators; add elements to these accumulators, merge the created
accumulators, and extract the output.

This can be performed by replacing the {{Combine.perKey}} composite transform with a multi-step
combine {{CombineBundles -> GroupByKey -> MergeAccumulators}}

Where {{CombineBundles}} is a {{ParDo}} which takes input {{KV<K, InputT>}} and produces
{{KV<K, AccumT>}}, outputting in {{FinishBundle}} (this can only be performed if the
Combine takes no side inputs or does not have merging windows). {{MergeAccumulators}} takes
in {{KV<K, Iterable<AccumT>>}} and produces {{KV<K, OutputT>}} by merging
all of the accumulators and extracting the output.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message