calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Chauhan <hashut...@apache.org>
Subject Re: Adding Exchange operator and Distribution trait
Date Tue, 17 Feb 2015 23:14:18 GMT
One thing I found missing which was there in original proposal was a class
combining both sorting & partitioning traits, SortExchange. It will be
useful to have that.

On Tue, Feb 17, 2015 at 2:25 PM, Ashutosh Chauhan <hashutosh@apache.org>
wrote:

> Looks good on the first look. Once we have more experience after
> integrating in Hive, I may have further feedback.
>
> On Fri, Feb 13, 2015 at 12:39 AM, Julian Hyde <jhyde@apache.org> wrote:
>
>> I have committed, on my branch
>> https://github.com/julianhyde/incubator-calcite/tree/calcite-594, a fix
>> for
>> https://issues.apache.org/jira/browse/CALCITE-594 that I think meets
>> Drill's and Ashutosh's requirements. Please review it before I commit to
>> master.
>>
>> I have 3 questions:
>>
>> 1. Can we please shorten the distribution type name HASH_DISTRIBUTED to
>> HASH (similarly RANDOM_DISTRIBUTED etc.)?
>>
>> 2. Drill has DrillDistributionTrait.DEFAULT, but I'd rather not create
>> RelDistributions.DEFAULT. (The default will be different for different
>> systems using Calcite.)
>>
>> 3. RelDistributionTraitDef.convert does not create specific physical
>> operators as it does in Drill, because there are no physical
>> implementations of Exchange, just LogicalExchange. Drill will need to
>> create rules to convert LogicalExchange to HashToRandomExchangePrel, etc.
>> Hope that is OK.
>>
>> Julian
>>
>>
>>
>> On Feb 12, 2015, at 1:57 PM, Jacques Nadeau <jacques@apache.org> wrote:
>>
>> Got it.
>>
>> On Thu, Feb 12, 2015 at 11:44 AM, Julian Hyde <julian@hydromatic.net>
>> wrote:
>>
>> RelMdDistribution (the "Md" that distinguishes it from
>> "RelDistribution" stands for "metadata") is just theory, but it will
>> look a lot like RelMdCollation.
>>
>>
>>
>> https://github.com/apache/incubator-calcite/blob/master/core/src/main/java/org/apache/calcite/rel/metadata/RelMdCollation.java
>>
>> On Wed, Feb 11, 2015 at 6:47 PM, Ashutosh Chauhan <hashutosh@apache.org>
>> wrote:
>>
>> I think RelDistribution as a member of Exchange (as in Julian proposal)
>> makes more sense as compared to mine at field level.
>>
>> On Wed, Feb 11, 2015 at 6:42 PM, Jacques Nadeau <jacques@apache.org>
>>
>> wrote:
>>
>>
>> Does RelMdDistribution exist or as theory?
>>
>> Definitely inclined to "Exchange" over "Shuffle" as more
>>
>> generic/consistent
>>
>> with Volcano.
>>
>> Note, looking at Ashutosh I wonder a little about the need to have range
>> versus hash at the field information level.  I'm sure we can come up
>>
>> with a
>>
>> use case but it seems to overcomplicate.  We already have space
>>
>> exploration
>>
>> problems when trying to do distribution subsumption for joins with trait
>> propagation so I would be cautious about adding more dimensions to that
>> attribute.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message