spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DB Tsai <>
Subject Re: MLLib - Thoughts about refactoring Updater for LBFGS?
Date Fri, 07 Mar 2014 00:42:24 GMT
On Thu, Mar 6, 2014 at 4:26 PM, David Hall <> wrote:
> I'm not sure why Spark should be serializing LBFGS? Shouldn't it live on
> the controller node? Or is this a per-node thing?
> But no problem to make it serializable.
It will live in the controller node. Only RDD operations are per-node
thing. I'm calling RDD operations inside the DiffFunction, so Spark
should not serialize anything. But it seems that Spark serialize evey
parent objects; can Spark anyone confirm this?

> Err, yeah. I should probably have LBFGS do this automatically, but there's
> a CachedDiffFunction that gets rid of the redundant calculations.
It will nice to have LBFHS do this automatically. Just try
CachedDiffFunction, and it works.


DB Tsai
Machine Learning Engineer
Alpine Data Labs

View raw message