spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ayan guha <guha.a...@gmail.com>
Subject Re: Spark ML - Is IDF model reusable
Date Tue, 01 Nov 2016 23:09:50 GMT
Yes, that is correct. I think I misread a part of it in terms of
scoring....I think we both are saying same thing so thats a good thing :)

On Wed, Nov 2, 2016 at 10:04 AM, Nirav Patel <npatel@xactlycorp.com> wrote:

> Hi Ayan,
>
> "classification algorithm will for sure need to Fit against new dataset
> to produce new model" I said this in context of re-training the model. Is
> it not correct? Isn't it part of re-training?
>
> Thanks
>
> On Tue, Nov 1, 2016 at 4:01 PM, ayan guha <guha.ayan@gmail.com> wrote:
>
>> Hi
>>
>> "classification algorithm will for sure need to Fit against new dataset
>> to produce new model" - I do not think this is correct. Maybe we are
>> talking semantics but AFAIU, you "train" one model using some dataset, and
>> then use it for scoring new datasets.
>>
>> You may re-train every month, yes. And you may run cross validation once
>> a month (after re-training) or lower freq like once in 2-3 months to
>> validate model quality. Here, number of months are not important, but you
>> must be running cross validation and similar sort of "model evaluation"
>> work flow typically in lower frequency than Re-Training process.
>>
>> On Wed, Nov 2, 2016 at 5:48 AM, Nirav Patel <npatel@xactlycorp.com>
>> wrote:
>>
>>> Hi Ayan,
>>> After deployment, we might re-train it every month. That is whole
>>> different problem I have explored yet. classification algorithm will for
>>> sure need to Fit against new dataset to produce new model. Correct me if I
>>> am wrong but I think I will also FIt new IDF model based on new dataset. At
>>> that time as well I will follow same training-validation split (or
>>> corss-validation) to evaluate model performance on new data before
>>> releasing it to make prediction. So afik , every time you  need to re-train
>>> model you will need to corss validate using some data split strategy.
>>>
>>> I think spark ML document should start explaining mathematical model or
>>> simple algorithm what Fit and Transform means for particular algorithm
>>> (IDF, NaiveBayes)
>>>
>>> Thanks
>>>
>>> On Tue, Nov 1, 2016 at 5:45 AM, ayan guha <guha.ayan@gmail.com> wrote:
>>>
>>>> I have come across similar situation recently and decided to run
>>>> Training  workflow less frequently than scoring workflow.
>>>>
>>>> In your use case I would imagine you will run IDF fit workflow once in
>>>> say a week. It will produce a model object which will be saved. In scoring
>>>> workflow, you will typically see new unseen dataset and the model generated
>>>> in training flow will be used to score or label this new dataset.
>>>>
>>>> Note, train and test datasets are used during development phase when
>>>> you are trying to find out which model to use and
>>>> efficientcy/performance/accuracy etc. It will never be part of
>>>> workflow. In a little elaborate setting you may want to automate model
>>>> evaluations, but that's a different story.
>>>>
>>>> Not sure if I could explain properly, please feel free to comment.
>>>> On 1 Nov 2016 22:54, "Nirav Patel" <npatel@xactlycorp.com> wrote:
>>>>
>>>>> Yes, I do apply NaiveBayes after IDF .
>>>>>
>>>>> " you can re-train (fit) on all your data before applying it to
>>>>> unseen data." Did you mean I can reuse that model to Transform both
>>>>> training and test data?
>>>>>
>>>>> Here's the process:
>>>>>
>>>>> Datasets:
>>>>>
>>>>>    1. Full sample data (labeled)
>>>>>    2. Training (labeled)
>>>>>    3. Test (labeled)
>>>>>    4. Unseen (non-labeled)
>>>>>
>>>>> Here are two workflow options I see:
>>>>>
>>>>> Option - 1 (currently using)
>>>>>
>>>>>    1. Fit IDF model (idf-1) on full Sample data
>>>>>    2. Apply(Transform) idf-1 on full sample data
>>>>>    3. Split data set into Training and Test data
>>>>>    4. Fit ML model on Training data
>>>>>    5. Apply(Transform) model on Test data
>>>>>    6. Apply(Transform) idf-1 on Unseen data
>>>>>    7. Apply(Transform) model on Unseen data
>>>>>
>>>>> Option - 2
>>>>>
>>>>>    1. Split sample data into Training and Test data
>>>>>    2. Fit IDF model (idf-1) only on training data
>>>>>    3. Apply(Transform) idf-1 on training data
>>>>>    4. Apply(Transform) idf-1 on test data
>>>>>    5. Fit ML model on Training data
>>>>>    6. Apply(Transform) model on Test data
>>>>>    7. Apply(Transform) idf-1 on Unseen data
>>>>>    8. Apply(Transform) model on Unseen data
>>>>>
>>>>> So you are suggesting Option-2 in this particular case, right?
>>>>>
>>>>> On Tue, Nov 1, 2016 at 4:24 AM, Robin East <robin.east@xense.co.uk>
>>>>> wrote:
>>>>>
>>>>>> Fit it on training data to evaluate the model. You can either use
>>>>>> that model to apply to unseen data or you can re-train (fit) on all
your
>>>>>> data before applying it to unseen data.
>>>>>>
>>>>>> fit and transform are 2 different things: fit creates a model,
>>>>>> transform applies a model to data to create transformed output. If
you are
>>>>>> using your training data in a subsequent step (e.g. running logistic
>>>>>> regression or some other machine learning algorithm) then you need
to
>>>>>> transform your training data using the IDF model before passing it
through
>>>>>> the next step.
>>>>>>
>>>>>> ------------------------------------------------------------
>>>>>> -------------------
>>>>>> Robin East
>>>>>> *Spark GraphX in Action* Michael Malak and Robin East
>>>>>> Manning Publications Co.
>>>>>> http://www.manning.com/books/spark-graphx-in-action
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 1 Nov 2016, at 11:18, Nirav Patel <npatel@xactlycorp.com>
wrote:
>>>>>>
>>>>>> Just to re-iterate what you said, I should fit IDF model only on
>>>>>> training data and then re-use it for both test data and then later
on
>>>>>> unseen data to make predictions.
>>>>>>
>>>>>> On Tue, Nov 1, 2016 at 3:49 AM, Robin East <robin.east@xense.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>> The point of setting aside a portion of your data as a test set
is
>>>>>>> to try and mimic applying your model to unseen data. If you fit
your IDF
>>>>>>> model to all your data, any evaluation you perform on your test
set is
>>>>>>> likely to over perform compared to ‘real’ unseen data. Effectively
you
>>>>>>> would have overfit your model.
>>>>>>> ------------------------------------------------------------
>>>>>>> -------------------
>>>>>>> Robin East
>>>>>>> *Spark GraphX in Action* Michael Malak and Robin East
>>>>>>> Manning Publications Co.
>>>>>>> http://www.manning.com/books/spark-graphx-in-action
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 1 Nov 2016, at 10:15, Nirav Patel <npatel@xactlycorp.com>
wrote:
>>>>>>>
>>>>>>> FYI, I do reuse IDF model while making prediction against new
>>>>>>> unlabeled data but not between training and test data while training
a
>>>>>>> model.
>>>>>>>
>>>>>>> On Tue, Nov 1, 2016 at 3:10 AM, Nirav Patel <npatel@xactlycorp.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I am using IDF estimator/model (TF-IDF) to convert text features
>>>>>>>> into vectors. Currently, I fit IDF model on all sample data
and then
>>>>>>>> transform them. I read somewhere that I should split my data
into training
>>>>>>>> and test before fitting IDF model; Fit IDF only on training
data and then
>>>>>>>> use same transformer to transform training and test data.
>>>>>>>> This raise more questions:
>>>>>>>> 1) Why would you do that? What exactly do IDF learn during
fitting
>>>>>>>> process that it can reuse to transform any new dataset. Perhaps
idea is to
>>>>>>>> keep same value for |D| and DF|t, D| while use new TF|t,
D| ?
>>>>>>>> 2) If not then fitting and transforming seems redundant for
IDF
>>>>>>>> model
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [image: What's New with Xactly]
>>>>>>> <http://www.xactlycorp.com/email-click/>
>>>>>>>
>>>>>>> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
>>>>>>> <https://www.linkedin.com/company/xactly-corporation> 
[image:
>>>>>>> Twitter] <https://twitter.com/Xactly>  [image: Facebook]
>>>>>>> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
>>>>>>> <http://www.youtube.com/xactlycorporation>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [image: What's New with Xactly]
>>>>>> <http://www.xactlycorp.com/email-click/>
>>>>>>
>>>>>> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
>>>>>> <https://www.linkedin.com/company/xactly-corporation>  [image:
>>>>>> Twitter] <https://twitter.com/Xactly>  [image: Facebook]
>>>>>> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
>>>>>> <http://www.youtube.com/xactlycorporation>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [image: What's New with Xactly]
>>>>> <http://www.xactlycorp.com/email-click/>
>>>>>
>>>>> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
>>>>> <https://www.linkedin.com/company/xactly-corporation>  [image:
>>>>> Twitter] <https://twitter.com/Xactly>  [image: Facebook]
>>>>> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
>>>>> <http://www.youtube.com/xactlycorporation>
>>>>
>>>>
>>>
>>>
>>>
>>> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>>>
>>> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
>>> <https://www.linkedin.com/company/xactly-corporation>  [image: Twitter]
>>> <https://twitter.com/Xactly>  [image: Facebook]
>>> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
>>> <http://www.youtube.com/xactlycorporation>
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>
>
>
>
> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>
> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
> <https://www.linkedin.com/company/xactly-corporation>  [image: Twitter]
> <https://twitter.com/Xactly>  [image: Facebook]
> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
> <http://www.youtube.com/xactlycorporation>
>



-- 
Best Regards,
Ayan Guha

Mime
View raw message