calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Chauhan <hashut...@apache.org>
Subject Re: Adding Exchange operator and Distribution trait
Date Tue, 17 Feb 2015 22:25:50 GMT
Looks good on the first look. Once we have more experience after
integrating in Hive, I may have further feedback.

On Fri, Feb 13, 2015 at 12:39 AM, Julian Hyde <jhyde@apache.org> wrote:

> I have committed, on my branch
> https://github.com/julianhyde/incubator-calcite/tree/calcite-594, a fix
> for
> https://issues.apache.org/jira/browse/CALCITE-594 that I think meets
> Drill's and Ashutosh's requirements. Please review it before I commit to
> master.
>
> I have 3 questions:
>
> 1. Can we please shorten the distribution type name HASH_DISTRIBUTED to
> HASH (similarly RANDOM_DISTRIBUTED etc.)?
>
> 2. Drill has DrillDistributionTrait.DEFAULT, but I'd rather not create
> RelDistributions.DEFAULT. (The default will be different for different
> systems using Calcite.)
>
> 3. RelDistributionTraitDef.convert does not create specific physical
> operators as it does in Drill, because there are no physical
> implementations of Exchange, just LogicalExchange. Drill will need to
> create rules to convert LogicalExchange to HashToRandomExchangePrel, etc.
> Hope that is OK.
>
> Julian
>
>
>
> On Feb 12, 2015, at 1:57 PM, Jacques Nadeau <jacques@apache.org> wrote:
>
> Got it.
>
> On Thu, Feb 12, 2015 at 11:44 AM, Julian Hyde <julian@hydromatic.net>
> wrote:
>
> RelMdDistribution (the "Md" that distinguishes it from
> "RelDistribution" stands for "metadata") is just theory, but it will
> look a lot like RelMdCollation.
>
>
>
> https://github.com/apache/incubator-calcite/blob/master/core/src/main/java/org/apache/calcite/rel/metadata/RelMdCollation.java
>
> On Wed, Feb 11, 2015 at 6:47 PM, Ashutosh Chauhan <hashutosh@apache.org>
> wrote:
>
> I think RelDistribution as a member of Exchange (as in Julian proposal)
> makes more sense as compared to mine at field level.
>
> On Wed, Feb 11, 2015 at 6:42 PM, Jacques Nadeau <jacques@apache.org>
>
> wrote:
>
>
> Does RelMdDistribution exist or as theory?
>
> Definitely inclined to "Exchange" over "Shuffle" as more
>
> generic/consistent
>
> with Volcano.
>
> Note, looking at Ashutosh I wonder a little about the need to have range
> versus hash at the field information level.  I'm sure we can come up
>
> with a
>
> use case but it seems to overcomplicate.  We already have space
>
> exploration
>
> problems when trying to do distribution subsumption for joins with trait
> propagation so I would be cautious about adding more dimensions to that
> attribute.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message