spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Debasish Das <debasish.da...@gmail.com>
Subject Constraint Solver for Spark
Date Thu, 05 Jun 2014 22:38:58 GMT
Hi,

We are adding a constrained ALS solver in Spark to solve matrix
factorization use-cases which needs additional constraints (bounds,
equality, inequality, quadratic constraints)

We are using a native version of a primal dual SOCP solver due to its small
memory footprint and sparse ccs matrix computation it uses...The solver
depends on AMD and LDL packages from Timothy Davis for sparse ccs matrix
algebra (released under lgpl)...

Due to GPL dependencies, it won't be possible to release the code as Apache
license for now...If we get good results on our use-cases, we will plan to
write a version in breeze/modify joptimizer for sparse ccs operations...

I derived ConstrainedALS from Spark mllib ALS and I am comparing the
performance with default ALS and non-negative ALS as baseline. Plan is to
release the code as GPL license for community review...I have kept the
package structure as org.apache.spark.mllib.recommendation

There are some private functions defined in ALS, which I would like to
reuse....Is it possible to take the private out from the following
functions:

1. makeLinkRDDs
2. makeInLinkBlock
3. makeOutLinkBlock
4. randomFactor
5. unblockFactors

I don't want to copy any code.... I can ask for a PR to make these
changes...

Thanks.
Deb

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message