spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Cheung <>
Subject Re: [Spark R] Proposal: Exposing RBackend in RRunner
Date Wed, 28 Mar 2018 16:02:33 GMT
I think the difference is py4j is a public library whereas the R backend is specific to SparkR.

Can you elaborate what you need JVMObjectTracker for? We have provided R convenient APIs to
call into JVM: sparkR.callJMethod for example

From: Jeremy Liu <>
Sent: Tuesday, March 27, 2018 12:20 PM
Subject: Re: [Spark R] Proposal: Exposing RBackend in RRunner
To: <>

Spark Dev,

On second thought, the below topic seems more appropriate for spark-dev rather than spark-users:

Spark Users,

In SparkR, RBackend is created in RRunner.main(). This in particular makes it difficult to
control or use the RBackend. For my use case, I am looking to access the JVMObjectTracker
that RBackend maintains for SparkR dataframes.

Analogously, pyspark starts a py4j.GatewayServer in PythonRunner.main(). It's then possible
to start a ClientServer that then has access to the object bindings between Python/Java.

Is there something similar for SparkR? Or a reasonable way to expose RBackend?

Jeremy Liu<>

View raw message