spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Albert Manyà <alber...@eml.cc>
Subject Re: Serialize mllib's MatrixFactorizationModel
Date Mon, 15 Dec 2014 17:09:51 GMT
In that case, what is the strategy to train a model in some background
batch process and make recommendations for some other service in real
time? Run both processes in the same spark cluster?

Thanks.

-- 
  Albert Manyà
  albertmp@eml.cc

On Mon, Dec 15, 2014, at 05:58 PM, Sean Owen wrote:
> This class is not going to be serializable, as it contains huge RDDs.
> Even if the right constructor existed the RDDs inside would not
> serialize.
> 
> On Mon, Dec 15, 2014 at 4:33 PM, Albert Manyà <albertmp@eml.cc> wrote:
> > Hi all.
> >
> > I'm willing to serialize and later load a model trained using mllib's
> > ALS.
> >
> > I've tried usign Java serialization with something like:
> >
> >     val model = ALS.trainImplicit(training, rank, numIter, lambda, 1)
> >     val fos = new FileOutputStream("model.bin")
> >     val oos = new ObjectOutputStream(fos)
> >     oos.writeObject(bestModel.get)
> >
> > But when I try to deserialize it using:
> >
> >     val fos = new FileInputStream("model.bin")
> >     val oos = new ObjectInputStream(fos)
> >     val model = oos.readObject().asInstanceOf[MatrixFactorizationModel]
> >
> >  I get the error:
> >
> > Exception in thread "main" java.io.IOException: PARSING_ERROR(2)
> >
> > I've also tried to serialize MatrixFactorizationModel's both RDDs
> > (products and users) and later create the MatrixFactorizationModel by
> > hand passing the RDDs by constructor but I get an error cause its
> > private:
> >
> > Error:(58, 17) constructor MatrixFactorizationModel in class
> > MatrixFactorizationModel cannot be accessed in object RecommendALS
> >     val model = new MatrixFactorizationModel (8, userFeatures,
> >     productFeatures)
> >
> > Any ideas?
> >
> > Thanks!
> >
> > --
> >   Albert Manyà
> >   albertmp@eml.cc
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message