spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xi Shen <davidshe...@gmail.com>
Subject Re: How to reuse a ML trained model?
Date Sun, 08 Mar 2015 10:27:29 GMT
errr...do you have any suggestions for me before 1.3 release?

I can't believe there's no ML model serialize method in Spark. I think
training the models are quite expensive, isn't it?


Thanks,
David


On Sun, Mar 8, 2015 at 5:14 AM Burak Yavuz <brkyvz@gmail.com> wrote:

> Hi,
>
> There is model import/export for some of the ML algorithms on the current
> master (and they'll be shipped with the 1.3 release).
>
> Burak
> On Mar 7, 2015 4:17 AM, "Xi Shen" <davidshen84@gmail.com> wrote:
>
>> Wait...it seem SparkContext does not provide a way to save/load object
>> files. It can only save/load RDD. What do I missed here?
>>
>>
>> Thanks,
>> David
>>
>>
>> On Sat, Mar 7, 2015 at 11:05 PM Xi Shen <davidshen84@gmail.com> wrote:
>>
>>> Ah~it is serializable. Thanks!
>>>
>>>
>>> On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy <ekremaksoy@gmail.com>
>>> wrote:
>>>
>>>> You can serialize your trained model to persist somewhere.
>>>>
>>>> Ekrem Aksoy
>>>>
>>>> On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen <davidshen84@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I checked a few ML algorithms in MLLib.
>>>>>
>>>>> https://spark.apache.org/docs/0.8.1/api/mllib/index.html#
>>>>> org.apache.spark.mllib.classification.LogisticRegressionModel
>>>>>
>>>>> I could not find a way to save the trained model. Does this means I
>>>>> have to train my model every time? Is there a more economic way to do
this?
>>>>>
>>>>> I am thinking about something like:
>>>>>
>>>>> model.run(...)
>>>>> model.save("hdfs://path/to/hdfs")
>>>>>
>>>>> Then, next I can do:
>>>>>
>>>>> val model = Model.createFrom("hdfs://...")
>>>>> model.predict(vector)
>>>>>
>>>>> I am new to spark, maybe there are other ways to persistent the model?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>>
>>>>

Mime
View raw message