spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject Re: combine rdds?
Date Tue, 28 Oct 2014 00:17:54 GMT
this requires evaluation of the rdd to do the count.

val x: RDD[X] = ...
val y: RDD[X] = ...
x.cache
val z = if(x.count < thres) x.union(y) else x
On Oct 27, 2014 7:51 PM, "Josh J" <joshjdevl@gmail.com> wrote:

Hi,

How could I combine rdds? I would like to combine two RDDs if the count in
an RDD is not above some threshold.

Thanks,
Josh

Mime
View raw message