spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Imberman <daniel.imber...@gmail.com>
Subject Re: Creating a python port for a Scala Spark Projeect
Date Thu, 23 Jun 2016 02:15:08 GMT
Thank you Holden, I look forward to watching your talk!

On Wed, Jun 22, 2016 at 7:12 PM Holden Karau <holden@pigscanfly.ca> wrote:

> PySpark RDDs are (on the Java side) are essentially RDD of pickled objects
> and mostly (but not entirely) opaque to the JVM. It is possible (by using
> some internals) to pass a PySpark DataFrame to a Scala library (you may or
> may not find the talk I gave at Spark Summit useful
> https://www.youtube.com/watch?v=V6DkTVvy9vk as well as some of the Python
> examples in
> https://github.com/high-performance-spark/high-performance-spark-examples
> ). Good luck! :)
>
> On Wed, Jun 22, 2016 at 7:07 PM, Daniel Imberman <
> daniel.imberman@gmail.com> wrote:
>
>> Hi All,
>>
>> I've developed a spark module in scala that I would like to add a python
>> port for. I want to be able to allow users to create a pyspark RDD and send
>> it to my system. I've been looking into the pyspark source code as well as
>> py4J and was wondering if there has been anything like this implemented
>> before.
>>
>> Thank you
>>
>
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
>

Mime
View raw message