spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zuhair Khayyat <zuhair.khay...@gmail.com>
Subject Re: [GitHub] incubator-spark pull request: Adding RDD unique self cross product
Date Thu, 13 Feb 2014 08:51:44 GMT
I am using rdd.selfCartesian for optimization purposes. I am using Spark
for large data analytic project on relational data. My application
sometimes require to compare the table with itself looking for
inconsistency within the data regardless of the order of compared tuples.

One advantage of rdd.selfCartesian of is that it generates almost half
the results of rdd.cartesian(rdd). For example, a table with 100 rows,
the rdd.cartesian(rdd)
will generate 10000 tuples to compare while the rdd.selfCartesian will only
generate 5050 tuples.

Another advantage is that rdd.selfCartesian helps me to get rid of the
duplicate errors when searching for tuple inconsistencies. In
my application, if an error can be found for tuples with the order (tx,ty),
the same error can also be found if they are in the opposite order (ty,tx).
If I used rdd.cartesian(rdd) I will have to look for duplicate errors in
the resulted RDDPair and remove them.

Regards,
Zuhair Khayyat


On Wed, Feb 12, 2014 at 11:52 PM, rxin <git@git.apache.org> wrote:

> Github user rxin commented on the pull request:
>
>
> https://github.com/apache/incubator-spark/pull/587#issuecomment-34916300
>
>     Thanks for submitting this. Just curious, what is the advantage of
> this over rdd.cartesian(rdd), i.e. just use cartesian to join itself?
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message