spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DB Tsai <dbt...@alpinenow.com>
Subject Re: MLLib - Thoughts about refactoring Updater for LBFGS?
Date Fri, 07 Mar 2014 00:21:35 GMT
Hi David,

I can converge to the same result with your breeze LBFGS and Fortran
implementations now. Probably, I made some mistakes when I tried
breeze before. I apologize that I claimed it's not stable.

See the test case in BreezeLBFGSSuite.scala
https://github.com/AlpineNow/spark/tree/dbtsai-breezeLBFGS

This is training multinomial logistic regression against iris dataset,
and both optimizers can train the models with 98% training accuracy.

There are two issues to use Breeze in Spark,

1) When the gradientSum and lossSum are computed distributively in
custom defined DiffFunction which will be passed into your optimizer,
Spark will complain LBFGS class is not serializable. In
BreezeLBFGS.scala, I've to convert RDD to array to make it work
locally. It should be easy to fix by just having LBFGS to implement
Serializable.

2) Breeze computes redundant gradient and loss. See the following log
from both Fortran and Breeze implementations.

Thanks.

Fortran:
Iteration -1: loss 1.3862943611198926, diff 1.0
Iteration 0: loss 1.5846343143210866, diff 0.14307193024217352
Iteration 1: loss 1.1242501524477688, diff 0.29053004039012126
Iteration 2: loss 1.0930151243303563, diff 0.027782962952189336
Iteration 3: loss 1.054036932835569, diff 0.03566113127440601
Iteration 4: loss 0.9907956302751622, diff 0.05999907649459571
Iteration 5: loss 0.9184205380342829, diff 0.07304737423337761
Iteration 6: loss 0.8259870936519937, diff 0.10064381175132982
Iteration 7: loss 0.6327447552109574, diff 0.23395293458364716
Iteration 8: loss 0.5534101162436359, diff 0.1253815427665277
Iteration 9: loss 0.4045020086612566, diff 0.26907321376758075
Iteration 10: loss 0.3078824990823728, diff 0.23885980452569627

Breeze:
Iteration -1: loss 1.3862943611198926, diff 1.0
Mar 6, 2014 3:59:11 PM com.github.fommil.netlib.BLAS <clinit>
WARNING: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemBLAS
Mar 6, 2014 3:59:11 PM com.github.fommil.netlib.BLAS <clinit>
WARNING: Failed to load implementation from:
com.github.fommil.netlib.NativeRefBLAS
Iteration 0: loss 1.3862943611198926, diff 0.0
Iteration 1: loss 1.5846343143210866, diff 0.14307193024217352
Iteration 2: loss 1.1242501524477688, diff 0.29053004039012126
Iteration 3: loss 1.1242501524477688, diff 0.0
Iteration 4: loss 1.1242501524477688, diff 0.0
Iteration 5: loss 1.0930151243303563, diff 0.027782962952189336
Iteration 6: loss 1.0930151243303563, diff 0.0
Iteration 7: loss 1.0930151243303563, diff 0.0
Iteration 8: loss 1.054036932835569, diff 0.03566113127440601
Iteration 9: loss 1.054036932835569, diff 0.0
Iteration 10: loss 1.054036932835569, diff 0.0
Iteration 11: loss 0.9907956302751622, diff 0.05999907649459571
Iteration 12: loss 0.9907956302751622, diff 0.0
Iteration 13: loss 0.9907956302751622, diff 0.0
Iteration 14: loss 0.9184205380342829, diff 0.07304737423337761
Iteration 15: loss 0.9184205380342829, diff 0.0
Iteration 16: loss 0.9184205380342829, diff 0.0
Iteration 17: loss 0.8259870936519939, diff 0.1006438117513297
Iteration 18: loss 0.8259870936519939, diff 0.0
Iteration 19: loss 0.8259870936519939, diff 0.0
Iteration 20: loss 0.6327447552109576, diff 0.233952934583647
Iteration 21: loss 0.6327447552109576, diff 0.0
Iteration 22: loss 0.6327447552109576, diff 0.0
Iteration 23: loss 0.5534101162436362, diff 0.12538154276652747
Iteration 24: loss 0.5534101162436362, diff 0.0
Iteration 25: loss 0.5534101162436362, diff 0.0
Iteration 26: loss 0.40450200866125635, diff 0.2690732137675816
Iteration 27: loss 0.40450200866125635, diff 0.0
Iteration 28: loss 0.40450200866125635, diff 0.0
Iteration 29: loss 0.30788249908237314, diff 0.23885980452569502

Sincerely,

DB Tsai
Machine Learning Engineer
Alpine Data Labs
--------------------------------------
Web: http://alpinenow.com/


On Wed, Mar 5, 2014 at 2:00 PM, David Hall <dlwh@cs.berkeley.edu> wrote:
> On Wed, Mar 5, 2014 at 1:57 PM, DB Tsai <dbtsai@alpinenow.com> wrote:
>
>> Hi David,
>>
>> On Tue, Mar 4, 2014 at 8:13 PM, dlwh <david.lw.hall@gmail.com> wrote:
>> > I'm happy to help fix any problems. I've verified at points that the
>> > implementation gives the exact same sequence of iterates for a few
>> different
>> > functions (with a particular line search) as the c port of lbfgs. So I'm
>> a
>> > little surprised it fails where Fortran succeeds... but only a little.
>> This
>> > was fixed late last year.
>> I'm working on a reproducible test case using breeze vs fortran
>> implementation to show the problem I've run into. The test will be in
>> one of the test cases in my Spark fork, is it okay for you to
>> investigate the issue? Or do I need to make it as a standalone test?
>>
>
>
> Um, as long as it wouldn't be too hard to pull out.
>
>
>>
>> Will send you the test later today.
>>
>> Thanks.
>>
>> Sincerely,
>>
>> DB Tsai
>> Machine Learning Engineer
>> Alpine Data Labs
>> --------------------------------------
>> Web: http://alpinenow.com/
>>

Mime
View raw message