spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Franz Chien <franzj...@gmail.com>
Subject Can I group elements in RDD into different groups and let each group share some elements?‏
Date Sat, 02 May 2015 19:45:23 GMT
Hi all,

Can I group elements in RDD into different groups and let each group share
elements? For example, I have 10,000 elements in RDD from e1 to e10000, and
I want to group and aggregate them by another mapping with size of 2000,
ex: ( (e1,e42), (e1,e554), (e3, e554)…… (2000th group))

My first approach was to filter the RDD with mapping rules for 2000 times,
and then union them together. However, it ran forever. Does SPARK provide a
way to group elements in RDD like this please?


Thanks,


Franz

Mime
View raw message