spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xi Shen <davidshe...@gmail.com>
Subject Re: How to reuse a ML trained model?
Date Sat, 07 Mar 2015 12:15:05 GMT
Wait...it seem SparkContext does not provide a way to save/load object
files. It can only save/load RDD. What do I missed here?


Thanks,
David


On Sat, Mar 7, 2015 at 11:05 PM Xi Shen <davidshen84@gmail.com> wrote:

> Ah~it is serializable. Thanks!
>
>
> On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy <ekremaksoy@gmail.com> wrote:
>
>> You can serialize your trained model to persist somewhere.
>>
>> Ekrem Aksoy
>>
>> On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen <davidshen84@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I checked a few ML algorithms in MLLib.
>>>
>>> https://spark.apache.org/docs/0.8.1/api/mllib/index.html#
>>> org.apache.spark.mllib.classification.LogisticRegressionModel
>>>
>>> I could not find a way to save the trained model. Does this means I have
>>> to train my model every time? Is there a more economic way to do this?
>>>
>>> I am thinking about something like:
>>>
>>> model.run(...)
>>> model.save("hdfs://path/to/hdfs")
>>>
>>> Then, next I can do:
>>>
>>> val model = Model.createFrom("hdfs://...")
>>> model.predict(vector)
>>>
>>> I am new to spark, maybe there are other ways to persistent the model?
>>>
>>>
>>> Thanks,
>>> David
>>>
>>>
>>

Mime
View raw message