calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <jh...@apache.org>
Subject Re: Adding Exchange operator and Distribution trait
Date Wed, 18 Feb 2015 01:42:52 GMT
I’ve added SortExchange and checked in as
https://github.com/apache/incubator-calcite/commit/18d4f70a7ebd5f5b3b0d8b83820f8d770be1d19e.

Julian



On Tue, Feb 17, 2015 at 4:00 PM, Julian Hyde <jhyde@apache.org> wrote:
> Good catch. I meant to add SortExchange, but it slipped my mind. I will add
> it.
>
> Julian
>
> On Feb 17, 2015, at 3:14 PM, Ashutosh Chauhan <hashutosh@apache.org> wrote:
>
> One thing I found missing which was there in original proposal was a class
> combining both sorting & partitioning traits, SortExchange. It will be
> useful to have that.
>
> On Tue, Feb 17, 2015 at 2:25 PM, Ashutosh Chauhan <hashutosh@apache.org>
> wrote:
>
> Looks good on the first look. Once we have more experience after
> integrating in Hive, I may have further feedback.
>
> On Fri, Feb 13, 2015 at 12:39 AM, Julian Hyde <jhyde@apache.org> wrote:
>
> I have committed, on my branch
> https://github.com/julianhyde/incubator-calcite/tree/calcite-594, a fix
> for
> https://issues.apache.org/jira/browse/CALCITE-594 that I think meets
> Drill's and Ashutosh's requirements. Please review it before I commit to
> master.
>
> I have 3 questions:
>
> 1. Can we please shorten the distribution type name HASH_DISTRIBUTED to
> HASH (similarly RANDOM_DISTRIBUTED etc.)?
>
> 2. Drill has DrillDistributionTrait.DEFAULT, but I'd rather not create
> RelDistributions.DEFAULT. (The default will be different for different
> systems using Calcite.)
>
> 3. RelDistributionTraitDef.convert does not create specific physical
> operators as it does in Drill, because there are no physical
> implementations of Exchange, just LogicalExchange. Drill will need to
> create rules to convert LogicalExchange to HashToRandomExchangePrel, etc.
> Hope that is OK.
>
> Julian
>
>
>
> On Feb 12, 2015, at 1:57 PM, Jacques Nadeau <jacques@apache.org> wrote:
>
> Got it.
>
> On Thu, Feb 12, 2015 at 11:44 AM, Julian Hyde <julian@hydromatic.net>
> wrote:
>
> RelMdDistribution (the "Md" that distinguishes it from
> "RelDistribution" stands for "metadata") is just theory, but it will
> look a lot like RelMdCollation.
>
>
>
> https://github.com/apache/incubator-calcite/blob/master/core/src/main/java/org/apache/calcite/rel/metadata/RelMdCollation.java
>
> On Wed, Feb 11, 2015 at 6:47 PM, Ashutosh Chauhan <hashutosh@apache.org>
> wrote:
>
> I think RelDistribution as a member of Exchange (as in Julian proposal)
> makes more sense as compared to mine at field level.
>
> On Wed, Feb 11, 2015 at 6:42 PM, Jacques Nadeau <jacques@apache.org>
>
> wrote:
>
>
> Does RelMdDistribution exist or as theory?
>
> Definitely inclined to "Exchange" over "Shuffle" as more
>
> generic/consistent
>
> with Volcano.
>
> Note, looking at Ashutosh I wonder a little about the need to have range
> versus hash at the field information level.  I'm sure we can come up
>
> with a
>
> use case but it seems to overcomplicate.  We already have space
>
> exploration
>
> problems when trying to do distribution subsumption for joins with trait
> propagation so I would be cautious about adding more dimensions to that
> attribute.
>
>
>
>

Mime
View raw message