spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Using R code as part of a Spark Application
Date Wed, 29 Jun 2016 17:25:42 GMT
Still you need sparkR
> On 29 Jun 2016, at 19:14, John Aherne <john.aherne@justenough.com> wrote:
> 
> Microsoft Azure has an option to create a spark cluster with R Server. MS bought RevoScale
(I think that was the name) and just recently deployed it. 
> 
>> On Wed, Jun 29, 2016 at 10:53 AM, Xinh Huynh <xinh.huynh@gmail.com> wrote:
>> There is some new SparkR functionality coming in Spark 2.0, such as "dapply". You
could use SparkR to load a Parquet file and then run "dapply" to apply a function to each
partition of a DataFrame.
>> 
>> Info about loading Parquet file:
>> http://people.apache.org/~pwendell/spark-releases/spark-2.0.0-rc1-docs/sparkr.html#from-data-sources
>> 
>> API doc for "dapply":
>> http://people.apache.org/~pwendell/spark-releases/spark-2.0.0-rc1-docs/api/R/index.html
>> 
>> Xinh
>> 
>>> On Wed, Jun 29, 2016 at 6:54 AM, sujeet jog <sujeet.jog@gmail.com> wrote:
>>> try Spark pipeRDD's , you can invoke the R script from pipe , push  the stuff
you want to do on the Rscript stdin,  p 
>>> 
>>> 
>>>> On Wed, Jun 29, 2016 at 7:10 PM, Gilad Landau <Gilad.Landau@clicktale.com>
wrote:
>>>> Hello,
>>>> 
>>>>  
>>>> 
>>>> I want to use R code as part of spark application (the same way I would do
with Scala/Python).  I want to be able to run an R syntax as a map function on a big Spark
dataframe loaded from a parquet file.
>>>> 
>>>> Is this even possible or the only way to use R is as part of RStudio orchestration
of our Spark  cluster?
>>>> 
>>>>  
>>>> 
>>>> Thanks for the help!
>>>> 
>>>>  
>>>> 
>>>> Gilad
>>>> 
> 
> 
> 
> -- 
> John Aherne
> Big Data and SQL Developer
> 
> 
> 
> Cell:
> Email:
> Skype:
> Web:
> 
> +1 (303) 809-9718
> john.aherne@justenough.com
> john.aherne.je
> www.justenough.com
> 
> 
> Confidentiality Note: The information contained in this email and document(s) attached
are for the exclusive use of the addressee and may contain confidential, privileged and non-disclosable
information. If the recipient of this email is not the addressee, such recipient is strictly
prohibited from reading, photocopying, distribution or otherwise using this email or its contents
in any way.

Mime
View raw message