beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-5413) Add method for defining composite transforms as lambda expressions
Date Tue, 18 Sep 2018 14:41:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-5413?focusedWorklogId=145307&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145307
]

ASF GitHub Bot logged work on BEAM-5413:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 18/Sep/18 14:40
            Start Date: 18/Sep/18 14:40
    Worklog Time Spent: 10m 
      Work Description: jklukas commented on issue #6414: [BEAM-5413] Add PTransform::compose
for lambda-based composite transforms
URL: https://github.com/apache/beam/pull/6414#issuecomment-422421164
 
 
   I see now that docs are still in a separate repo. I'm backing out the changes to the getting-started
guide, but the content I'd propose adding to the Composite Transforms section is:
   
   ```
   {:.language-java}
   If your logic only needs to be used once, you may instead define your composite
   transform as an anonymous Java 8 lambda function passed to `PTransform.compose`:
   
   
    ```java
     input.apply(PTransform.compose((PCollection<String> lines) -> {
         PCollection<String> words = lines.apply(
             ParDo.of(new ExtractWordsFn()));
         PCollection<KV<String, Long>> wordCounts =
             words.apply(Count.<String>perElement());
         return wordCounts;
     }))
   \```
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 145307)
    Time Spent: 20m  (was: 10m)

> Add method for defining composite transforms as lambda expressions
> ------------------------------------------------------------------
>
>                 Key: BEAM-5413
>                 URL: https://issues.apache.org/jira/browse/BEAM-5413
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-core
>            Reporter: Jeff Klukas
>            Assignee: Kenneth Knowles
>            Priority: Minor
>             Fix For: 2.8.0
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Defining a composite transform today requires writing a full named subclass of PTransform
(as [the programming guide documents|https://beam.apache.org/documentation/programming-guide/#composite-transforms]
but there are cases where users may want to define a fairly trivial composite transform using
a less verbose Java 8 lambda expression.
> Consider an example where the user has defined MyDeserializationTransform that attempts
to deserialize byte arrays into some object, returning a PCollectionTuple  with tags for
successfully deserialized records (mainTag) and for errors (errorTag).
> If we introduce a PTransform::compose method that takes in a SerializableFunction, the
user can handle errors in a small lambda expression:
>  
> {code:java}
> byteArrays
>     .apply("attempt to deserialize messages", 
>         new MyDeserializationTransform())
>     .apply("write deserialization errors",
>         PTransform.compose((PCollectionTuple input) -> {
>             input
>               .get(errorTag)
>               .apply(new MyErrorOutputTransform());
>             return input.get(mainTag);
>         })
>     .apply("more processing on the deserialized messages", 
>          new MyOtherTransform())
> {code}
> This style allows a more concise and fluent pipeline definition than is currently possible.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message