spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eahlberg <eahlb...@gmail.com>
Subject Optimizing cartesian product using keys
Date Mon, 29 Feb 2016 16:56:23 GMT
Hello,

To avoid computing all possible combinations, I'm trying to group values
according to a certain key, and then compute the cartesian product of the
values for each key, i.e.:

Input
 [(k1, [v1]), (k1, [v2]), (k2, [v3])]

Desired output:
[(v1, v1), (v1, v2), (v2, v2), (v2, v1), (v3, v3)]

Currently I'm doing it as follows (product is from Python itertools):
input = sc.textFile('data.csv')
rdd = input.map(lambda x: (x.key, [x]))
rdd2 = rdd.reduceByKey(lambda x, y: x + y)
rdd3 = rdd2.flatMapValues(lambda x: itertools.product(x, x))
result = rdd3.map(lambda x: x[1])

This works fine for very small files, but when the list is of length ~1000
the computation completely freezes.

Thanks in advance!




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Optimizing-cartesian-product-using-keys-tp26361.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message