calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <jh...@apache.org>
Subject Re: Adding Exchange operator and Distribution trait
Date Wed, 18 Feb 2015 00:00:13 GMT
Good catch. I meant to add SortExchange, but it slipped my mind. I will add
it.

Julian

On Feb 17, 2015, at 3:14 PM, Ashutosh Chauhan <hashutosh@apache.org> wrote:

One thing I found missing which was there in original proposal was a class
combining both sorting & partitioning traits, SortExchange. It will be
useful to have that.

On Tue, Feb 17, 2015 at 2:25 PM, Ashutosh Chauhan <hashutosh@apache.org>
wrote:

Looks good on the first look. Once we have more experience after
integrating in Hive, I may have further feedback.

On Fri, Feb 13, 2015 at 12:39 AM, Julian Hyde <jhyde@apache.org> wrote:

I have committed, on my branch
https://github.com/julianhyde/incubator-calcite/tree/calcite-594, a fix
for
https://issues.apache.org/jira/browse/CALCITE-594 that I think meets
Drill's and Ashutosh's requirements. Please review it before I commit to
master.

I have 3 questions:

1. Can we please shorten the distribution type name HASH_DISTRIBUTED to
HASH (similarly RANDOM_DISTRIBUTED etc.)?

2. Drill has DrillDistributionTrait.DEFAULT, but I'd rather not create
RelDistributions.DEFAULT. (The default will be different for different
systems using Calcite.)

3. RelDistributionTraitDef.convert does not create specific physical
operators as it does in Drill, because there are no physical
implementations of Exchange, just LogicalExchange. Drill will need to
create rules to convert LogicalExchange to HashToRandomExchangePrel, etc.
Hope that is OK.

Julian



On Feb 12, 2015, at 1:57 PM, Jacques Nadeau <jacques@apache.org> wrote:

Got it.

On Thu, Feb 12, 2015 at 11:44 AM, Julian Hyde <julian@hydromatic.net>
wrote:

RelMdDistribution (the "Md" that distinguishes it from
"RelDistribution" stands for "metadata") is just theory, but it will
look a lot like RelMdCollation.



https://github.com/apache/incubator-calcite/blob/master/core/src/main/java/org/apache/calcite/rel/metadata/RelMdCollation.java

On Wed, Feb 11, 2015 at 6:47 PM, Ashutosh Chauhan <hashutosh@apache.org>
wrote:

I think RelDistribution as a member of Exchange (as in Julian proposal)
makes more sense as compared to mine at field level.

On Wed, Feb 11, 2015 at 6:42 PM, Jacques Nadeau <jacques@apache.org>

wrote:


Does RelMdDistribution exist or as theory?

Definitely inclined to "Exchange" over "Shuffle" as more

generic/consistent

with Volcano.

Note, looking at Ashutosh I wonder a little about the need to have range
versus hash at the field information level.  I'm sure we can come up

with a

use case but it seems to overcomplicate.  We already have space

exploration

problems when trying to do distribution subsumption for joins with trait
propagation so I would be cautious about adding more dimensions to that
attribute.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message