spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Burak Yavuz <brk...@gmail.com>
Subject Re: How to reuse a ML trained model?
Date Sat, 07 Mar 2015 18:14:53 GMT
Hi,

There is model import/export for some of the ML algorithms on the current
master (and they'll be shipped with the 1.3 release).

Burak
On Mar 7, 2015 4:17 AM, "Xi Shen" <davidshen84@gmail.com> wrote:

> Wait...it seem SparkContext does not provide a way to save/load object
> files. It can only save/load RDD. What do I missed here?
>
>
> Thanks,
> David
>
>
> On Sat, Mar 7, 2015 at 11:05 PM Xi Shen <davidshen84@gmail.com> wrote:
>
>> Ah~it is serializable. Thanks!
>>
>>
>> On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy <ekremaksoy@gmail.com> wrote:
>>
>>> You can serialize your trained model to persist somewhere.
>>>
>>> Ekrem Aksoy
>>>
>>> On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen <davidshen84@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I checked a few ML algorithms in MLLib.
>>>>
>>>> https://spark.apache.org/docs/0.8.1/api/mllib/index.html#
>>>> org.apache.spark.mllib.classification.LogisticRegressionModel
>>>>
>>>> I could not find a way to save the trained model. Does this means I
>>>> have to train my model every time? Is there a more economic way to do this?
>>>>
>>>> I am thinking about something like:
>>>>
>>>> model.run(...)
>>>> model.save("hdfs://path/to/hdfs")
>>>>
>>>> Then, next I can do:
>>>>
>>>> val model = Model.createFrom("hdfs://...")
>>>> model.predict(vector)
>>>>
>>>> I am new to spark, maybe there are other ways to persistent the model?
>>>>
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>>
>>>

Mime
View raw message