spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Bradley <jos...@databricks.com>
Subject Re: SVD in pyspark ?
Date Mon, 26 Jan 2015 20:23:13 GMT
Hi Andreas,

There unfortunately is not a Python API yet for distributed matrices or
their operations.  Here's the JIRA to follow to stay up-to-date on it:
https://issues.apache.org/jira/browse/SPARK-3956

There are internal wrappers (used to create the Python API), but they are
not really public APIs.  The bigger challenge is creating/storing the
distributed matrix in Python.

Joseph

On Sun, Jan 25, 2015 at 11:32 AM, Chip Senkbeil <chip.senkbeil@gmail.com>
wrote:

> Hi Andreas,
>
> With regard to the notebook interface,  you can use the Spark Kernel (
> https://github.com/ibm-et/spark-kernel) as the backend for an IPython 3.0
> notebook. The kernel is designed to be the foundation for interactive
> applications connecting to Apache Spark and uses the IPython 5.0 message
> protocol - used by IPython 3.0 - to communicate.
>
> See the getting started section here:
> https://github.com/ibm-et/spark-kernel/wiki/Getting-Started-with-the-Spark-Kernel
>
> It discusses getting IPython connected to a Spark Kernel. If you have any
> more questions, feel free to ask!
>
> Signed,
> Chip Senkbeil
> IBM Emerging Technologies Software Engineer
>
> On Sun Jan 25 2015 at 1:12:32 PM Andreas Rhode <m.a.rhode@gmail.com>
> wrote:
>
>> Is the distributed SVD functionality exposed to Python yet?
>>
>> Seems it's only available to scala or java, unless I am missing something,
>> looking for a pyspark equivalent to
>> org.apache.spark.mllib.linalg.SingularValueDecomposition
>>
>> In case it's not there yet, is there a way to make a wrapper to call from
>> python into the corresponding java/scala code? The reason for using python
>> instead of just directly  scala is that I like to take advantage of the
>> notebook interface for visualization.
>>
>> As a side, is there a inotebook like interface for the scala based REPL?
>>
>> Thanks
>>
>> Andreas
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.
>> 1001560.n3.nabble.com/SVD-in-pyspark-tp21356.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>

Mime
View raw message