spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kristina Rogale Plazonic <kpl...@gmail.com>
Subject Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?
Date Tue, 25 Aug 2015 18:06:56 GMT
YES PLEASE!

:)))))))

On Tue, Aug 25, 2015 at 1:57 PM, Burak Yavuz <brkyvz@gmail.com> wrote:

> Hmm. I have a lot of code on the local linear algebra operations using
> Spark's Matrix and Vector representations
> done for https://issues.apache.org/jira/browse/SPARK-6442.
>
> I can make a Spark package with that code if people are interested.
>
> Best,
> Burak
>
> On Tue, Aug 25, 2015 at 10:54 AM, Kristina Rogale Plazonic <
> kplazo@gmail.com> wrote:
>
>> However I do think it's easier than it seems to write the implicits;
>>> it doesn't involve new classes or anything. Yes it's pretty much just
>>> what you wrote. There is a class "Vector" in Spark. This declaration
>>> can be in an object; you don't implement your own class. (Also you can
>>> use "toBreeze" to get Breeze vectors.)
>>
>>
>> The implicit conversion with the implicit def happens for the first
>> vector in the sum, but not the second vector (see below).
>>
>> At this point I give up, because I spent way too much time.  I am so
>> disappointed.  So many times I heard "Spark makes simple things easy and
>> complicated things possible". Well, here is the simplest thing you can
>> imagine in linear algebra, but heck, it is not easy or intuitive.  It was
>> easier to run a DeepLearning algo (from another library) than add two
>> vectors.
>>
>> If anybody has a workaround other than implementing your own
>> add/substract/scalarMultiply, PLEASE let me know.
>>
>> Here is the code and error from (freshly started) spark-shell:
>>
>> scala> import breeze.linalg.{DenseVector => BDV, SparseVector => BSV,
>> Vector => BV}
>> import breeze.linalg.{DenseVector=>BDV, SparseVector=>BSV, Vector=>BV}
>>
>> scala> import org.apache.spark.mllib.linalg.Vectors
>> import org.apache.spark.mllib.linalg.Vectors
>>
>> scala> val v1 = Vectors.dense(1.0, 2.0, 3.0)
>> v1: org.apache.spark.mllib.linalg.Vector = [1.0,2.0,3.0]
>>
>> scala> import org.apache.spark.mllib.linalg.{Vector =>SparkVector}
>> import org.apache.spark.mllib.linalg.{Vector=>SparkVector}
>>
>> scala> object MyUtils {
>>      |   implicit def toBreeze(v:SparkVector) = BV(v.toArray)
>>      | }
>> warning: there were 1 feature warning(s); re-run with -feature for details
>> defined module MyUtils
>>
>> scala> import MyUtils._
>> import MyUtils._
>>
>> scala> v1:BV[Double]
>> res2: breeze.linalg.Vector[Double] = DenseVector(1.0, 2.0, 3.0)
>>
>> scala> v1 + v1
>> <console>:30: error: could not find implicit value for parameter op:
>> breeze.linalg.operators.OpAdd.Impl2[breeze.linalg.Vector[Double],org.apache.spark.mllib.linalg.Vector,That]
>>               v1 + v1
>>                  ^
>>
>>
>>
>
>

Mime
View raw message