hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mostafa Mokhtar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-10369) CBO: Don't use HiveDefaultCostModel when With Tez and hive.cbo.costmodel.extended enabled
Date Thu, 16 Apr 2015 22:46:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-10369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498904#comment-14498904
] 

Mostafa Mokhtar commented on HIVE-10369:
----------------------------------------

[~jcamachorodriguez]

> CBO: Don't use HiveDefaultCostModel when With Tez and hive.cbo.costmodel.extended enabled

> ------------------------------------------------------------------------------------------
>
>                 Key: HIVE-10369
>                 URL: https://issues.apache.org/jira/browse/HIVE-10369
>             Project: Hive
>          Issue Type: Sub-task
>          Components: CBO
>    Affects Versions: 1.2.0
>            Reporter: Mostafa Mokhtar
>            Assignee: Laljo John Pullokkaran
>             Fix For: 1.2.0
>
>
> When calculating parallelism, we end up using  HiveDefaultCostModel. getSplitCount which
returns null instead of  HiveOnTezCostModel.getSplitCount which results in wrong parallelism.
> This happens for this join 
> {code}
> org.apache.calcite.plan.RelOptUtil.toString(join)
> 	 (java.lang.String) HiveJoin(condition=[=($1, $3)], joinType=[inner], algorithm=[none],
cost=[not available])
>   HiveProject(cs_sold_date_sk=[$0], cs_bill_customer_sk=[$3], cs_sales_price=[$21])
>     HiveTableScan(table=[[tpcds_bin_orc_200.catalog_sales]])
>   HiveJoin(condition=[=($1, $2)], joinType=[inner], algorithm=[MapJoin], cost=[{2400000.0
rows, 6.400008E11 cpu, 1294.6098 io}])
>     HiveProject(c_customer_sk=[$0], c_current_addr_sk=[$4])
>       HiveTableScan(table=[[tpcds_bin_orc_200.customer]])
>     HiveProject(ca_address_sk=[$0], ca_state=[$8], ca_zip=[$9])
>       HiveTableScan(table=[[tpcds_bin_orc_200.customer_address]])
> {code}
> The issue appears to be happening very early when calling 
> {code}
> if (pushDownTree != null) {
>       costPushDown =
>           RelMetadataQuery.getCumulativeCost(pushDownTree.getJoinTree());
>     }
> {code}
> As pushDownTree.getJoinTree().joinAlgorithm = HiveOnTezCostModel$TezMapJoinAlgorithm
> Call stack.
> {code}
> HiveDefaultCostModel$DefaultJoinAlgorithm.getSplitCount(HiveJoin) line: 114	
> HiveJoin.getSplitCount() line: 136	
> HiveRelMdParallelism.splitCount(HiveJoin) line: 63	
> NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native
method]	
> NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> ReflectiveRelMetadataProvider$1$1.invoke(Object, Method, Object[]) line: 182	
> $Proxy46.splitCount() line: not available	
> GeneratedMethodAccessor26.invoke(Object, Object[]) line: not available	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(Object, Method, Object[])
line: 109	
> $Proxy46.splitCount() line: not available	
> GeneratedMethodAccessor26.invoke(Object, Object[]) line: not available	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(Object, Method, Object[])
line: 109	
> $Proxy46.splitCount() line: not available	
> GeneratedMethodAccessor26.invoke(Object, Object[]) line: not available	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> CachingRelMetadataProvider$CachingInvocationHandler.invoke(Object, Method, Object[])
line: 132	
> $Proxy46.splitCount() line: not available	
> RelMetadataQuery.splitCount(RelNode) line: 401	
> HiveOnTezCostModel$TezMapJoinAlgorithm.getCost(HiveJoin) line: 255	
> HiveOnTezCostModel(HiveCostModel).getJoinCost(HiveJoin) line: 64	
> HiveRelMdCost.getNonCumulativeCost(HiveJoin) line: 56	
> NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native
method]	
> NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> ReflectiveRelMetadataProvider$1$1.invoke(Object, Method, Object[]) line: 182	
> $Proxy41.getNonCumulativeCost() line: not available	
> GeneratedMethodAccessor22.invoke(Object, Object[]) line: not available	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(Object, Method, Object[])
line: 109	
> $Proxy41.getNonCumulativeCost() line: not available	
> GeneratedMethodAccessor22.invoke(Object, Object[]) line: not available	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(Object, Method, Object[])
line: 109	
> $Proxy41.getNonCumulativeCost() line: not available	
> GeneratedMethodAccessor22.invoke(Object, Object[]) line: not available	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(Object, Method, Object[])
line: 109	
> $Proxy41.getNonCumulativeCost() line: not available	
> GeneratedMethodAccessor22.invoke(Object, Object[]) line: not available	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> CachingRelMetadataProvider$CachingInvocationHandler.invoke(Object, Method, Object[])
line: 132	
> $Proxy41.getNonCumulativeCost() line: not available	
> RelMetadataQuery.getNonCumulativeCost(RelNode) line: 115	
> HiveRelMdDistinctRowCount.getCumulativeCost(HiveJoin) line: 114	
> NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native
method]	
> NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> ReflectiveRelMetadataProvider$1$1.invoke(Object, Method, Object[]) line: 182	
> $Proxy40.getCumulativeCost() line: not available	
> GeneratedMethodAccessor21.invoke(Object, Object[]) line: not available	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(Object, Method, Object[])
line: 109	
> $Proxy40.getCumulativeCost() line: not available	
> GeneratedMethodAccessor21.invoke(Object, Object[]) line: not available	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(Object, Method, Object[])
line: 109	
> $Proxy40.getCumulativeCost() line: not available	
> GeneratedMethodAccessor21.invoke(Object, Object[]) line: not available	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> Method.invoke(Object, Object...) line: 606	
> CachingRelMetadataProvider$CachingInvocationHandler.invoke(Object, Method, Object[])
line: 132	
> $Proxy40.getCumulativeCost() line: not available	
> RelMetadataQuery.getCumulativeCost(RelNode) line: 101	
> LoptOptimizeJoinRule.addFactorToTree(LoptMultiJoin, LoptSemiJoinOptimizer, LoptJoinTree,
int, BitSet, List<RexNode>, boolean) line: 940	
> LoptOptimizeJoinRule.createOrdering(LoptMultiJoin, LoptSemiJoinOptimizer, int) line:
726	
> LoptOptimizeJoinRule.findBestOrderings(LoptMultiJoin, LoptSemiJoinOptimizer, RelOptRuleCall)
line: 458	
> LoptOptimizeJoinRule.onMatch(RelOptRuleCall) line: 128	
> HepPlanner(AbstractRelOptPlanner).fireRule(RelOptRuleCall) line: 326	
> HepPlanner.applyRule(RelOptRule, HepRelVertex, boolean) line: 515	
> HepPlanner.applyRules(Collection<RelOptRule>, boolean) line: 392	
> HepPlanner.executeInstruction(HepInstruction$RuleInstance) line: 255	
> HepInstruction$RuleInstance.execute(HepPlanner) line: 125	
> HepPlanner.executeProgram(HepProgram) line: 207	
> HepPlanner.findBestExp() line: 194	
> CalcitePlanner$CalcitePlannerAction.apply(RelOptCluster, RelOptSchema, SchemaPlus) line:
849	
> CalcitePlanner$CalcitePlannerAction.apply(RelOptCluster, RelOptSchema, SchemaPlus) line:
761	
> Frameworks$1.apply(RelOptCluster, RelOptSchema, SchemaPlus, CalciteServerStatement) line:
109	
> CalcitePrepareImpl.perform(CalciteServerStatement, PrepareAction<R>) line: 730

> Frameworks.withPrepare(PrepareAction<R>) line: 145	
> Frameworks.withPlanner(PlannerAction<R>, FrameworkConfig) line: 105	
> CalcitePlanner.getOptimizedAST() line: 602	
> CalcitePlanner.genOPTree(ASTNode, SemanticAnalyzer$PlannerContext) line: 240	
> CalcitePlanner(SemanticAnalyzer).analyzeInternal(ASTNode, SemanticAnalyzer$PlannerContext)
line: 10003	
> CalcitePlanner.analyzeInternal(ASTNode) line: 203	
> CalcitePlanner(BaseSemanticAnalyzer).analyze(ASTNode, Context) line: 224	
> ExplainSemanticAnalyzer.analyzeInternal(ASTNode) line: 74	
> ExplainSemanticAnalyzer(BaseSemanticAnalyzer).analyze(ASTNode, Context) line: 224	
> Driver.compile(String, boolean) line: 424	
> Driver.compile(String) line: 308	
> Driver.compileInternal(String) line: 1122	
> Driver.runInternal(String, boolean) line: 1170	
> Driver.run(String, boolean) line: 1059	
> Driver.run(String) line: 1049	
> CliDriver.processLocalCmd(String, CommandProcessor, CliSessionState) line: 213	
> CliDriver.processCmd(String) line: 165	
> CliDriver.processLine(String, boolean) line: 376	
> CliDriver.executeDriver(CliSessionState, HiveConf, OptionsProcessor) line: 736	
> CliDriver.run(String[]) line: 681	
> CliDriver.main(String[]) line: 621	
> NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native
method]	
> NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57	
> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message