spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tanveer Ahmad - EWI <>
Subject Spark dataframe creation through already distributed in-memory data sets
Date Tue, 16 Jun 2020 14:01:13 GMT
Hi all,

I am new to the Spark community. Please ignore if this question doesn't make sense.

My PySpark Dataframe is just taking a fraction of time (in ms) in 'Sorting', but moving data
is much expensive (> 14 sec).


I have a huge Arrow RecordBatches collection which is equally distributed on all of my worker
node's memories (in plasma_store). Currently, I am collecting all those RecordBatches back
in my master node, merging them, and converting them to a single Spark Dataframe. Then I apply
sorting function on that dataframe.

Spark dataframe is a cluster distributed data collection.

So my question is:

Is it possible to create a Spark dataframe from all that already distributed Arrow RecordBatches
data collections in the worker's nodes memories? So that the data should remain in the respective
worker's nodes memories (instead of bringing it to master node, merging, and then creating
distributed dataframe).


Tanveer Ahmad

View raw message