spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: RDD.join vs spark SQL join
Date Fri, 14 Aug 2015 09:21:56 GMT
Both works the same way, but with SparkSQL you will get the optimization
etc done by the catalyst. One important thing to consider is the #
partitions and the key distribution (when you are doing RDD.join), If the
keys are not evenly distributed across machines then you can see the
process chocking on a single task (more like it takes hell lot of time for
one task to execute compared to others in that stage).

Thanks
Best Regards

On Fri, Aug 14, 2015 at 1:25 AM, Xiao JIANG <jiangxiao01@outlook.com> wrote:

> Hi,
>
> May I know the performance difference the rdd.join function and spark SQL
> join operation. If I want to join several big Rdds, how should I decide
> which one I should use? What are the factors to consider here?
>
> Thanks!
>

Mime
View raw message