spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <nicholas.cham...@gmail.com>
Subject Persisting PySpark ML Pipelines that include custom Transformers
Date Fri, 19 Aug 2016 18:28:52 GMT
I understand persistence for PySpark ML pipelines is already present in
2.0, and further improvements are being made for 2.1 (e.g. SPARK-13786
<https://issues.apache.org/jira/browse/SPARK-13786>).

I’m having trouble, though, persisting a pipeline that includes a custom
Transformer (see SPARK-17025
<https://issues.apache.org/jira/browse/SPARK-17025>). It appears that there
is a magic _to_java() method that I need to implement.

Is the intention that developers implementing custom Transformers would
also specify how it should be persisted, or are there ideas about how to
make this automatic? I searched on JIRA but I’m not sure if I missed an
issue that already addresses this problem.

Nick
​

Mime
View raw message