beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <>
Subject [jira] [Commented] (BEAM-801) SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
Date Wed, 02 Nov 2016 18:23:59 GMT


Eugene Kirpichov commented on BEAM-801:

> SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
> --------------------------------------------------------------
>                 Key: BEAM-801
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-core, runner-direct
>            Reporter: Kenneth Knowles
>            Assignee: Eugene Kirpichov
> Today, runners-core contains an expansion of {{ParDo(DoFn)}} that implements it via unsupported
features of {{ParDo(OldDoFn)}}. This expansion is used by the DirectRunner.
> The right approach to provide a ready implementation in runners-core is the one taken
by {{GroupAlsoByWindowViaWindowSetDoFn}} where the unsupported features are provided to the
constructor of the {{DoFn}}, rather than expected to be passed through the {{ProcessContext}}.
> These features are not guaranteed to be part of the public state & timers API of
{{DoFn}}, particularly in the case of watermark holds, so it is not prudent to plan on waiting
for the new state & timers API and porting over.
> These issues create real blockers by causing dependency cycles between implementing {{DoFn}}
fully (requires no use of unsupported features), implementing new state and timers (requires
new {{DoFn}}), and keeping the hack until new {{DoFn}} has state and timers (requires use
of unsupported features).
> To break the cycle, the tests that rely on unsupported features are being disabled. They
can be re-enabled either by following the design suggested above (probably the most robust
approach) or by waiting and porting to new features as they are available.

This message was sent by Atlassian JIRA

View raw message