spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph K. Bradley (JIRA)" <>
Subject [jira] [Updated] (SPARK-7675) PySpark Params type conversions
Date Thu, 13 Aug 2015 01:29:46 GMT


Joseph K. Bradley updated SPARK-7675:
    Target Version/s: 1.6.0  (was: 1.5.0)

> PySpark Params type conversions
> ----------------------------------------
>                 Key: SPARK-7675
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML, PySpark
>            Reporter: Joseph K. Bradley
>            Priority: Minor
> Currently, PySpark wrappers for Scala classes are brittle when accepting Param
types.  E.g., Normalizer's "p" param cannot be set to "2" (an integer); it must be set to
"2.0" (a float).  Fixing this is not trivial since there does not appear to be a natural place
to insert the conversion before Python wrappers call Java's Params setter method.
> A possible fix will be to include a method "_checkType" to PySpark's Param class which
checks the type, prints an error if needed, and converts types when relevant (e.g., int to
float, or scipy matrix to array).  The Java wrapper method which copies params to Scala can
call this method when available.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message