beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-801) SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
Date Wed, 02 Nov 2016 18:23:59 GMT

    [ https://issues.apache.org/jira/browse/BEAM-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15629943#comment-15629943
] 

Eugene Kirpichov commented on BEAM-801:
---------------------------------------

https://github.com/apache/incubator-beam/pull/1261

> SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
> --------------------------------------------------------------
>
>                 Key: BEAM-801
>                 URL: https://issues.apache.org/jira/browse/BEAM-801
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-core, runner-direct
>            Reporter: Kenneth Knowles
>            Assignee: Eugene Kirpichov
>
> Today, runners-core contains an expansion of {{ParDo(DoFn)}} that implements it via unsupported
features of {{ParDo(OldDoFn)}}. This expansion is used by the DirectRunner.
> The right approach to provide a ready implementation in runners-core is the one taken
by {{GroupAlsoByWindowViaWindowSetDoFn}} where the unsupported features are provided to the
constructor of the {{DoFn}}, rather than expected to be passed through the {{ProcessContext}}.
> These features are not guaranteed to be part of the public state & timers API of
{{DoFn}}, particularly in the case of watermark holds, so it is not prudent to plan on waiting
for the new state & timers API and porting over.
> These issues create real blockers by causing dependency cycles between implementing {{DoFn}}
fully (requires no use of unsupported features), implementing new state and timers (requires
new {{DoFn}}), and keeping the hack until new {{DoFn}} has state and timers (requires use
of unsupported features).
> To break the cycle, the tests that rely on unsupported features are being disabled. They
can be re-enabled either by following the design suggested above (probably the most robust
approach) or by waiting and porting to new features as they are available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message