spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Liu1" <>
Subject what does dapply actually do?
Date Wed, 18 Jan 2017 19:30:23 GMT

I'm really new and trying to learn sparkR. I have defined a relatively
complicated user-defined function, and use dapply() to apply the function
on a SparkDataFrame. It was very fast. But I am not sure what has actually
been done by dapply(). Because when I used collect() to see the output,
which is very simple, it took a long time to get the result. I suppose
maybe I don't need to use collect(), but without using it, how can I output
the final results, say, in a .csv file?
Thank you very much for the help.

Best Regards,

From:	Ninad Shringarpure <>
To:	user <>
Date:	01/18/2017 02:24 PM
Subject:	Creating UUID using SparksSQL

Hi Team,

Is there a standard way of generating a unique id for each row in from
Spark SQL. I am looking for functionality similar to UUID generation in

Let me know if you need any additional information.


View raw message