drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aman Sinha" <asi...@maprtech.com>
Subject Re: Review Request 20741: Cost model to include distribution cost and enable new plans
Date Thu, 01 May 2014 17:16:13 GMT


> On April 29, 2014, 5:11 a.m., Jacques Nadeau wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamAggPrule.java,
line 80
> > <https://reviews.apache.org/r/20741/diff/1/?file=569068#file569068line80>
> >
> >     same as hash agg, why removed?
> 
> Aman Sinha wrote:
>     That was just for testing.. I have uncommented it.

I found an issue with single grouping key distribution for StreamingAggregate...some TPCH
tests timeout and seem to be stuck in VolcanoPlanner.canonize() method.  Note that the HashAggregate
with single grouping key distribution works ok.  One possibility is that for StreamingAggr
even if the distribution is on single key, the collation is on all keys, so that might be
causing some problem.  I will re-visit this later.  The current plans I am generating (and
comparing by cost) are: (1) HashAggr with distribution on all grouping keys, (2) HashAggr
with distribution on single grouping key, (3) StreamingAggr with distribution on all grouping
keys.  


- Aman


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20741/#review41681
-----------------------------------------------------------


On April 26, 2014, 1:54 a.m., Aman Sinha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20741/
> -----------------------------------------------------------
> 
> (Updated April 26, 2014, 1:54 a.m.)
> 
> 
> Review request for drill, Jacques Nadeau and Jinfeng Ni.
> 
> 
> Repository: drill-git
> 
> 
> Description
> -------
> 
> 1. Added DrillCostBase that includes distribution (network) cost.
> 2. Added cost formulas for computeSelfCost() for various Exchange operators, joins, aggregations
etc. 
> 3. Added Prels for HashJoin, HashAggregate and generate plans for those.  
> 4. Added exchange Prels: BroadcastExchangePrel and a new type called HashToMergeExchangePrel.

> 
> 
> Diffs
> -----
> 
>   exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 7e3b63d 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToMergeExchange.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableConfig.java
98892c0 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillFilterRelBase.java
955729b 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillProjectRelBase.java
cf3d188 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillScanRelBase.java
b370352 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillScreenRelBase.java
51ed442 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillCostBase.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelOptCost.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelOptCostFactory.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java
1492a28 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/BroadcastExchangePrel.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTrait.java
b75fb40 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTraitDef.java
c2ebb7a 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/FilterPrel.java
0fc3abd 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashAggPrel.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashAggPrule.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrel.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrule.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashToMergeExchangePrel.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashToRandomExchangePrel.java
e5c9661 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MergeJoinPrel.java
978a531 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MergeJoinPrule.java
8298e50 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
e6e99c0 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129

>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SingleMergeExchangePrel.java
d9431cc 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SortPrel.java 344be4e

>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamAggPrel.java
c2880da 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamAggPrule.java
a561a61 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/TopNPrel.java e981a45

>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/UnionExchangePrel.java
f89cbaa 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
8892a8f 
>   exec/java-exec/src/test/java/org/apache/drill/TestTpchSingleMode.java 1ccb65c 
>   exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoin.java
ba067e2 
>   exec/java-exec/src/test/resources/join/hj_exchanges.json PRE-CREATION 
>   exec/java-exec/src/test/resources/queries/tpch/01.sql f33416f 
> 
> Diff: https://reviews.apache.org/r/20741/diff/
> 
> 
> Testing
> -------
> 
> Existing unit tests.  Some TPCH tests show memory leaks but that is not directly related
to the costing changes...however those leaks would have to be resolved before enabling the
new plans.  However, I wanted to get this review request out to get feedback.  
> 
> 
> Thanks,
> 
> Aman Sinha
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message