spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DB Tsai <dbt...@alpinenow.com>
Subject Re: MLLib - Thoughts about refactoring Updater for LBFGS?
Date Fri, 07 Mar 2014 08:46:03 GMT
Hi Xiangrui,

I think it doesn't matter whether we use Fortran/Breeze/RISO for
optimizers since optimization only takes << 1% of time. Most of the
time is in gradientSum and lossSum parallel computation.

Sincerely,

DB Tsai
Machine Learning Engineer
Alpine Data Labs
--------------------------------------
Web: http://alpinenow.com/


On Thu, Mar 6, 2014 at 7:10 PM, Xiangrui Meng <mengxr@gmail.com> wrote:
> Hi DB,
>
> Thanks for doing the comparison! What were the running times for
> fortran/breeze/riso?
>
> Best,
> Xiangrui
>
> On Thu, Mar 6, 2014 at 4:21 PM, DB Tsai <dbtsai@alpinenow.com> wrote:
>> Hi David,
>>
>> I can converge to the same result with your breeze LBFGS and Fortran
>> implementations now. Probably, I made some mistakes when I tried
>> breeze before. I apologize that I claimed it's not stable.
>>
>> See the test case in BreezeLBFGSSuite.scala
>> https://github.com/AlpineNow/spark/tree/dbtsai-breezeLBFGS
>>
>> This is training multinomial logistic regression against iris dataset,
>> and both optimizers can train the models with 98% training accuracy.
>>
>> There are two issues to use Breeze in Spark,
>>
>> 1) When the gradientSum and lossSum are computed distributively in
>> custom defined DiffFunction which will be passed into your optimizer,
>> Spark will complain LBFGS class is not serializable. In
>> BreezeLBFGS.scala, I've to convert RDD to array to make it work
>> locally. It should be easy to fix by just having LBFGS to implement
>> Serializable.
>>
>> 2) Breeze computes redundant gradient and loss. See the following log
>> from both Fortran and Breeze implementations.
>>
>> Thanks.
>>
>> Fortran:
>> Iteration -1: loss 1.3862943611198926, diff 1.0
>> Iteration 0: loss 1.5846343143210866, diff 0.14307193024217352
>> Iteration 1: loss 1.1242501524477688, diff 0.29053004039012126
>> Iteration 2: loss 1.0930151243303563, diff 0.027782962952189336
>> Iteration 3: loss 1.054036932835569, diff 0.03566113127440601
>> Iteration 4: loss 0.9907956302751622, diff 0.05999907649459571
>> Iteration 5: loss 0.9184205380342829, diff 0.07304737423337761
>> Iteration 6: loss 0.8259870936519937, diff 0.10064381175132982
>> Iteration 7: loss 0.6327447552109574, diff 0.23395293458364716
>> Iteration 8: loss 0.5534101162436359, diff 0.1253815427665277
>> Iteration 9: loss 0.4045020086612566, diff 0.26907321376758075
>> Iteration 10: loss 0.3078824990823728, diff 0.23885980452569627
>>
>> Breeze:
>> Iteration -1: loss 1.3862943611198926, diff 1.0
>> Mar 6, 2014 3:59:11 PM com.github.fommil.netlib.BLAS <clinit>
>> WARNING: Failed to load implementation from:
>> com.github.fommil.netlib.NativeSystemBLAS
>> Mar 6, 2014 3:59:11 PM com.github.fommil.netlib.BLAS <clinit>
>> WARNING: Failed to load implementation from:
>> com.github.fommil.netlib.NativeRefBLAS
>> Iteration 0: loss 1.3862943611198926, diff 0.0
>> Iteration 1: loss 1.5846343143210866, diff 0.14307193024217352
>> Iteration 2: loss 1.1242501524477688, diff 0.29053004039012126
>> Iteration 3: loss 1.1242501524477688, diff 0.0
>> Iteration 4: loss 1.1242501524477688, diff 0.0
>> Iteration 5: loss 1.0930151243303563, diff 0.027782962952189336
>> Iteration 6: loss 1.0930151243303563, diff 0.0
>> Iteration 7: loss 1.0930151243303563, diff 0.0
>> Iteration 8: loss 1.054036932835569, diff 0.03566113127440601
>> Iteration 9: loss 1.054036932835569, diff 0.0
>> Iteration 10: loss 1.054036932835569, diff 0.0
>> Iteration 11: loss 0.9907956302751622, diff 0.05999907649459571
>> Iteration 12: loss 0.9907956302751622, diff 0.0
>> Iteration 13: loss 0.9907956302751622, diff 0.0
>> Iteration 14: loss 0.9184205380342829, diff 0.07304737423337761
>> Iteration 15: loss 0.9184205380342829, diff 0.0
>> Iteration 16: loss 0.9184205380342829, diff 0.0
>> Iteration 17: loss 0.8259870936519939, diff 0.1006438117513297
>> Iteration 18: loss 0.8259870936519939, diff 0.0
>> Iteration 19: loss 0.8259870936519939, diff 0.0
>> Iteration 20: loss 0.6327447552109576, diff 0.233952934583647
>> Iteration 21: loss 0.6327447552109576, diff 0.0
>> Iteration 22: loss 0.6327447552109576, diff 0.0
>> Iteration 23: loss 0.5534101162436362, diff 0.12538154276652747
>> Iteration 24: loss 0.5534101162436362, diff 0.0
>> Iteration 25: loss 0.5534101162436362, diff 0.0
>> Iteration 26: loss 0.40450200866125635, diff 0.2690732137675816
>> Iteration 27: loss 0.40450200866125635, diff 0.0
>> Iteration 28: loss 0.40450200866125635, diff 0.0
>> Iteration 29: loss 0.30788249908237314, diff 0.23885980452569502
>>
>> Sincerely,
>>
>> DB Tsai
>> Machine Learning Engineer
>> Alpine Data Labs
>> --------------------------------------
>> Web: http://alpinenow.com/
>>
>>
>> On Wed, Mar 5, 2014 at 2:00 PM, David Hall <dlwh@cs.berkeley.edu> wrote:
>>> On Wed, Mar 5, 2014 at 1:57 PM, DB Tsai <dbtsai@alpinenow.com> wrote:
>>>
>>>> Hi David,
>>>>
>>>> On Tue, Mar 4, 2014 at 8:13 PM, dlwh <david.lw.hall@gmail.com> wrote:
>>>> > I'm happy to help fix any problems. I've verified at points that the
>>>> > implementation gives the exact same sequence of iterates for a few
>>>> different
>>>> > functions (with a particular line search) as the c port of lbfgs. So
I'm
>>>> a
>>>> > little surprised it fails where Fortran succeeds... but only a little.
>>>> This
>>>> > was fixed late last year.
>>>> I'm working on a reproducible test case using breeze vs fortran
>>>> implementation to show the problem I've run into. The test will be in
>>>> one of the test cases in my Spark fork, is it okay for you to
>>>> investigate the issue? Or do I need to make it as a standalone test?
>>>>
>>>
>>>
>>> Um, as long as it wouldn't be too hard to pull out.
>>>
>>>
>>>>
>>>> Will send you the test later today.
>>>>
>>>> Thanks.
>>>>
>>>> Sincerely,
>>>>
>>>> DB Tsai
>>>> Machine Learning Engineer
>>>> Alpine Data Labs
>>>> --------------------------------------
>>>> Web: http://alpinenow.com/
>>>>

Mime
View raw message