spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiangrui Meng <>
Subject Re: Re-use scaling means and variances from StandardScalerModel
Date Sat, 10 Jan 2015 06:14:55 GMT
Feel free to create a JIRA for this issue. We might need to discuss
what to put in the public constructors. In the meanwhile, you can use
Java serialization to save/load the model:

sc.parallelize(Seq(model), 1).saveAsObjectFile("/tmp/model")
val model = sc.objectFile[StandardScalerModel]("/tmp/model").first()


On Fri, Jan 9, 2015 at 12:21 PM, ogeagla <> wrote:
> Hello,
> I would like to re-use the means and variances computed by the fit function
> in the StandardScaler, as I persist them and my use case requires consisted
> scaling of data based on some initial data set.  The StandardScalerModel's
> constructor takes means and variances, but is private[mllib].
> Forking/compiling Spark or copy/pasting the class into my project are both
> options, but  I'd like to stay away from them.  Any chance there is interest
> in a PR to allow this re-use via removal of private from the the
> constructor?  Or perhaps an alternative solution exists?
> Thanks,
> Octavian
> --
> View this message in context:
> Sent from the Apache Spark Developers List mailing list archive at
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message