beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Work logged] (BEAM-5413) Add method for defining composite transforms as lambda expressions
Date Thu, 20 Sep 2018 12:33:00 GMT


ASF GitHub Bot logged work on BEAM-5413:

                Author: ASF GitHub Bot
            Created on: 20/Sep/18 12:32
            Start Date: 20/Sep/18 12:32
    Worklog Time Spent: 10m 
      Work Description: jklukas commented on issue #6414: [BEAM-5413] Add PTransform::compose
for lambda-based composite transforms
   Just squashed in a change to add `@Experimental` annotation per mailing list discussion.

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 146002)
    Time Spent: 0.5h  (was: 20m)

> Add method for defining composite transforms as lambda expressions
> ------------------------------------------------------------------
>                 Key: BEAM-5413
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-core
>            Reporter: Jeff Klukas
>            Assignee: Kenneth Knowles
>            Priority: Minor
>             Fix For: 2.8.0
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
> Defining a composite transform today requires writing a full named subclass of PTransform
(as [the programming guide documents|]
but there are cases where users may want to define a fairly trivial composite transform using
a less verbose Java 8 lambda expression.
> Consider an example where the user has defined MyDeserializationTransform that attempts
to deserialize byte arrays into some object, returning a PCollectionTuple  with tags for
successfully deserialized records (mainTag) and for errors (errorTag).
> If we introduce a PTransform::compose method that takes in a SerializableFunction, the
user can handle errors in a small lambda expression:
> {code:java}
> byteArrays
>     .apply("attempt to deserialize messages", 
>         new MyDeserializationTransform())
>     .apply("write deserialization errors",
>         PTransform.compose((PCollectionTuple input) -> {
>             input
>               .get(errorTag)
>               .apply(new MyErrorOutputTransform());
>             return input.get(mainTag);
>         })
>     .apply("more processing on the deserialized messages", 
>          new MyOtherTransform())
> {code}
> This style allows a more concise and fluent pipeline definition than is currently possible.

This message was sent by Atlassian JIRA

View raw message