spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pbaier <>
Subject Behaviour of RDD sampling
Date Tue, 31 May 2016 07:45:09 GMT
Hi all,

I have to following use case:
I have around 10k of jsons that I want to use for learning.
The jsons are all stored in one file.

For learning a ML model, however, I only need around 30% of the jsons (the
rest is not needed at all).
So, my idea was to load all data into a RDD and then use the rdd.sample
method to get my fraction of the data.
I implemented this, and in the end it took as long as loading the whole data
So I was wondering if Spark is still loading the whole dataset from disk and
does the filtering afterwards?
If this is the case, why does Spark not push down the filtering and load
only a fraction of data from the disk?



View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message