spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiangrui Meng <men...@gmail.com>
Subject Re: Re-use scaling means and variances from StandardScalerModel
Date Sat, 10 Jan 2015 06:14:55 GMT
Feel free to create a JIRA for this issue. We might need to discuss
what to put in the public constructors. In the meanwhile, you can use
Java serialization to save/load the model:

sc.parallelize(Seq(model), 1).saveAsObjectFile("/tmp/model")
val model = sc.objectFile[StandardScalerModel]("/tmp/model").first()

-Xiangrui

On Fri, Jan 9, 2015 at 12:21 PM, ogeagla <ogeagla@gmail.com> wrote:
> Hello,
>
> I would like to re-use the means and variances computed by the fit function
> in the StandardScaler, as I persist them and my use case requires consisted
> scaling of data based on some initial data set.  The StandardScalerModel's
> constructor takes means and variances, but is private[mllib].
> Forking/compiling Spark or copy/pasting the class into my project are both
> options, but  I'd like to stay away from them.  Any chance there is interest
> in a PR to allow this re-use via removal of private from the the
> constructor?  Or perhaps an alternative solution exists?
>
> Thanks,
> Octavian
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Re-use-scaling-means-and-variances-from-StandardScalerModel-tp10073.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message