spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: map-side-combine in Spark SQL
Date Wed, 10 Feb 2016 21:08:33 GMT
I'm not 100% sure I understand your question, but yes, Spark (both the RDD
API and SQL/DataFrame) does partial aggregation.


On Tue, Feb 9, 2016 at 8:37 PM, Rishitesh Mishra <rishi80.mishra@gmail.com>
wrote:

> Can anybody confirm, whether ANY operator in Spark SQL uses
> map-side-combine ? If not, is it safe to assume SortShuffleManager will
> always use Serialized sorting in case of queries from Spark SQL ?
>

Mime
View raw message