spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Liu <jeremy.jl....@gmail.com>
Subject Re: [Spark R] Proposal: Exposing RBackend in RRunner
Date Thu, 29 Mar 2018 15:23:58 GMT
Use case is to cache a reference to the JVM object created by SparkR.

On Wed, Mar 28, 2018 at 12:03 PM Reynold Xin <rxin@databricks.com> wrote:

> If you need the functionality I would recommend you just copying the code
> over to your project and use it that way.
>
> On Wed, Mar 28, 2018 at 9:02 AM Felix Cheung <felixcheung_m@hotmail.com>
> wrote:
>
>> I think the difference is py4j is a public library whereas the R backend
>> is specific to SparkR.
>>
>> Can you elaborate what you need JVMObjectTracker for? We have provided R
>> convenient APIs to call into JVM: sparkR.callJMethod for example
>>
>> _____________________________
>> From: Jeremy Liu <jeremy.jl.liu@gmail.com>
>> Sent: Tuesday, March 27, 2018 12:20 PM
>> Subject: Re: [Spark R] Proposal: Exposing RBackend in RRunner
>> To: <dev@spark.apache.org>
>>
>>
>>
>> Spark Dev,
>>
>> On second thought, the below topic seems more appropriate for spark-dev
>> rather than spark-users:
>>
>> Spark Users,
>>>
>>> In SparkR, RBackend is created in RRunner.main(). This in particular
>>> makes it difficult to control or use the RBackend. For my use case, I am
>>> looking to access the JVMObjectTracker that RBackend maintains for SparkR
>>> dataframes.
>>>
>>> Analogously, pyspark starts a py4j.GatewayServer in PythonRunner.main().
>>> It's then possible to start a ClientServer that then has access to the
>>> object bindings between Python/Java.
>>>
>>> Is there something similar for SparkR? Or a reasonable way to expose
>>> RBackend?
>>>
>>> Thanks!
>>>
>> --
>> -----
>> Jeremy Liu
>> jeremy.jl.liu@gmail.com
>>
>>
>> --
-----
Jeremy Liu
jeremy.jl.liu@gmail.com

Mime
View raw message