spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Jin <ice.xell...@gmail.com>
Subject Re: Spark join over sorted columns of dataset.
Date Sun, 12 Mar 2017 17:54:15 GMT
I am not an expert on this but here is what I think:

Catalyst maintains information on whether a plan node is ordered. If your
dataframe is a result of a order by, catalyst will skip the sorting when it
does merge sort join. If you dataframe is created from storage, for
instance. ParquetRelation, then I am not sure if there is an API that
allows user to tell Catalyst that the ParquetRelation is ordered on column
x. If there isn't, it's probably useful to add.

Li
On Fri, Mar 3, 2017 at 11:23 AM Koert Kuipers <koert@tresata.com> wrote:

> For RDD the shuffle is already skipped but the sort is not. In
> spark-sorted we track partitioning and sorting within partitions for
> key-value RDDs and can avoid the sort. See:
> https://github.com/tresata/spark-sorted
>
> For Dataset/DataFrame such optimizations are done automatically, however
> it's currently not always working for Dataset, see:
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/SPARK-19468
>
> On Mar 3, 2017 11:06 AM, "Rohit Verma" <rohit.verma@rokittech.com> wrote:
>
> Sending it to dev’s.
> Can you please help me providing some ideas for below.
>
> Regards
> Rohit
> > On Feb 23, 2017, at 3:47 PM, Rohit Verma <rohit.verma@rokittech.com>
> wrote:
> >
> > Hi
> >
> > While joining two columns of different dataset, how to optimize join if
> both the columns are pre sorted within the dataset.
> > So that when spark do sort merge join the sorting phase can skipped.
> >
> > Regards
> > Rohit
>
>
>

Mime
View raw message