beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismaël Mejía (JIRA) <>
Subject [jira] [Assigned] (BEAM-797) A PipelineVisitor that creates a Spark-native pipeline.
Date Wed, 23 Nov 2016 10:10:01 GMT


Ismaël Mejía reassigned BEAM-797:

    Assignee: Ismaël Mejía

> A PipelineVisitor that creates a Spark-native pipeline. 
> --------------------------------------------------------
>                 Key: BEAM-797
>                 URL:
>             Project: Beam
>          Issue Type: Wish
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Ismaël Mejía
>            Priority: Minor
> It could be very useful for debugging purposes to have a custom PipelineVisitor that
can tell what's the underlying Spark code that is being called.
> One idea:
> This could be called with a flag in SparkPipelineOptions and instead of executing the
pipeline, it would print the underlying Spark DAG.
> Clearly, DoFn internals would be obfuscated, but the Spark code could note {{mapPartitions("ExtractWords")}}
> Another difference would be Sources as they are a custom implementation for Beam.

This message was sent by Atlassian JIRA

View raw message