Thanks Jules ! Appreciate it a lot indeed !




On Mon, Mar 19, 2018 at 7:16 PM, Jules Damji <dmatrix@comcast.net> wrote:
What’s your PySpark function? Is it a UDF? If so consider using pandas UDF introduced in Spark 2.3. 

More info here: https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html


Sent from my iPhone
Pardon the dumb thumb typos :)

On Mar 18, 2018, at 10:54 PM, Debabrata Ghosh <mailfordebu@gmail.com> wrote:

Hi,
             My dataframe is having 2000 rows. For processing each row it consider 3 seconds and so sequentially it takes 2000 * 3 = 6000 seconds , which is a very high time.

              Further, I am contemplating to run the function in parallel. For example, I would like to divide the total rows in my dataframe by 4 and accordingly I will prepare a set of 500 rows and want to call my pyspark function in parallel. I wanted to know if there is any library / pyspark function which I can leverage to do this execution in parallel.

               Will really appreciate for your feedback as per your earliest convenience. Thanks,

Debu