spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yanbo Liang <yblia...@gmail.com>
Subject Re: Spark ML - Java implementation of custom Transformer
Date Sat, 02 Jul 2016 07:23:38 GMT
Hi Mehdi,

Could you share your code and then we can help you to figure out the
problem?
Actually JavaTestParams can work well but there is some compatibility issue
for JavaDeveloperApiExample.
We have removed JavaDeveloperApiExample temporary at Spark 2.0 in order to
not confuse users. Since the solution for the compatibility issue has been
figured out, we will add it back at 2.1.

Thanks
Yanbo

2016-06-27 11:57 GMT-07:00 Mehdi Meziane <mehdi.meziane@ldmobile.net>:

> Hi all,
>
> We have some problems while implementing custom Transformers in JAVA
> (SPARK 1.6.1).
> We do override the method copy, but it crashes with an AbstractMethodError.
>
> If we extends the UnaryTransformer, and do not override the copy method,
> it works without any error.
>
> We tried to write the copy like in these examples :
>
> https://github.com/apache/spark/blob/branch-2.0/mllib/src/test/java/org/apache/spark/ml/param/JavaTestParams.java
>
> https://github.com/eBay/Spark/blob/branch-1.6/examples/src/main/java/org/apache/spark/examples/ml/JavaDeveloperApiExample.java
>
> None of it worked.
>
> The copy is defined in the Params class as :
>
>   /**
>    * Creates a copy of this instance with the same UID and some extra
> params.
>    * Subclasses should implement this method and set the return type
> properly.
>    *
>    * @see [[defaultCopy()]]
>    */
>   def copy(extra: ParamMap): Params
>
> Any idea?
> Thanks,
>
> Mehdi
>

Mime
View raw message