spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Bradley <jos...@databricks.com>
Subject Re: DAG in Pipeline
Date Sun, 12 Jun 2016 17:47:51 GMT
Hi Pranay,

Yes, you can do this.  The DAG structure should be specified via the
various Transformers' input and output columns, where a Transformer can
have multiple input and/or output columns.  Most of the classification and
regression Models are good examples of Transformers with multiple input and
output columns.

Hope this helps!
Joseph

On Wed, Jun 8, 2016 at 9:59 PM, Pranay Tonpay <ptonpay@gmail.com> wrote:

> Hi,
> Pipeline as of now seems to be having a series of transformers and
> estimators in a serial fashion.
> Is it possible to create a DAG sort of thing -
> Eg -
> Two transformers running in parallel to cleanse data (a custom built
> Transformer)  in some way and then their outputs ( two outputs ) used for
> some sort of correlation ( another custom built Transformer )
>
> Let me know -
>
> thx
> pranay
>

Mime
View raw message