spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roch Denis <>
Subject Re: Help in merging a RDD agaisnt itself using the V of a (K,V).
Date Thu, 24 Jul 2014 01:09:30 GMT
Ah yes, you're quite right with partitions I could probably process a good
chunk of the data but I didn't think a reduce would work? Sorry, I'm still
new to Spark and map reduce in general but I thought that the reduce result
wasn't an RDD and had to fit into memory. If the result of a reduce can be
any size, then yes I can see how to make it work. 

Sorry for not being certain, the doc is not quite clear on that point, at
least to me. 

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message