spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Liu1" <liux...@us.ibm.com>
Subject what does dapply actually do?
Date Wed, 18 Jan 2017 19:30:23 GMT

Hi,
I'm really new and trying to learn sparkR. I have defined a relatively
complicated user-defined function, and use dapply() to apply the function
on a SparkDataFrame. It was very fast. But I am not sure what has actually
been done by dapply(). Because when I used collect() to see the output,
which is very simple, it took a long time to get the result. I suppose
maybe I don't need to use collect(), but without using it, how can I output
the final results, say, in a .csv file?
Thank you very much for the help.

Best Regards,
Xiao




From:	Ninad Shringarpure <ninad@cloudera.com>
To:	user <user@spark.apache.org>
Date:	01/18/2017 02:24 PM
Subject:	Creating UUID using SparksSQL



Hi Team,

Is there a standard way of generating a unique id for each row in from
Spark SQL. I am looking for functionality similar to UUID generation in
hive.

Let me know if you need any additional information.

Thanks,
Ninad


Mime
View raw message