drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <tianxiang.c...@qubida.com>
Subject Issue with drill query
Date Tue, 07 Feb 2017 15:55:19 GMT
Hi all,

I hope this is the correct place to ask for help.

We have some hive tables stored as textfile that we try to query from, and perform some joins.
Because we have programmed a SQL Query generator and based on end user’s selection, it came
up with a SQL query that is about 1 million characters long and has about 700 selects/nested
queries.
After submitting the query thru Drill’s web interface, the query is accepted but stuck in
Starting state. Further checks from log file, we got the following. 
We are running a 2 node drill cluster, m4.2xlarge 8 vCPU and 32GB of ram.

Thus wanted to check if anyone has ran a long query like ours before or we should be looking
at another area on the issue.

Thank you in advance!

2017-02-07 15:25:45,995 [27661a58-2d7e-650c-0000-b1c82b38bb9a:foreman] ERROR o.a.d.e.s.hive.HiveMetadataProvider
- Failed to parse Hive stats in metastore.
java.lang.NumberFormatException: null
        at java.lang.Long.parseLong(Long.java:552) ~[na:1.8.0_111]
        at java.lang.Long.valueOf(Long.java:803) ~[na:1.8.0_111]
        at org.apache.drill.exec.store.hive.HiveMetadataProvider.getStatsFromProps(HiveMetadataProvider.java:212)
[drill-storage-hive-core-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.store.hive.HiveMetadataProvider.getStats(HiveMetadataProvider.java:90)
[drill-storage-hive-core-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.store.hive.HiveScan.getScanStats(HiveScan.java:224) [drill-storage-hive-core-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.physical.base.AbstractGroupScan.getScanStats(AbstractGroupScan.java:79)
[drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.planner.logical.DrillScanRel.computeSelfCost(DrillScanRel.java:159)
[drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.calcite.rel.metadata.RelMdPercentageOriginalRows.getNonCumulativeCost(RelMdPercentageOriginalRows.java:165)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_111]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
        at org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke(ReflectiveRelMetadataProvider.java:182)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at com.sun.proxy.$Proxy71.getNonCumulativeCost(Unknown Source) [na:na]
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_111]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
        at org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(ChainedRelMetadataProvider.java:109)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at com.sun.proxy.$Proxy71.getNonCumulativeCost(Unknown Source) [na:na]
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_111]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
        at org.apache.calcite.rel.metadata.CachingRelMetadataProvider$CachingInvocationHandler.invoke(CachingRelMetadataProvider.java:132)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at com.sun.proxy.$Proxy71.getNonCumulativeCost(Unknown Source) [na:na]
        at org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost(RelMetadataQuery.java:115)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.volcano.VolcanoPlanner.getCost(VolcanoPlanner.java:1112)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0(RelSubset.java:363)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements(RelSubset.java:344)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet(VolcanoPlanner.java:1827)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1760)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:1017)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1037)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1940)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:138)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:213) [calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch(DrillPushProjIntoScan.java:90)
[drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) [calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
        at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:404)
[drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:343)
[drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:240)
[drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:290)
[drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168)
[drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:123)
[drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:97)
[drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1008) [drill-java-exec-1.9.0.jar:1.9.0]
        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264) [drill-java-exec-1.9.0.jar:1.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

Kind Regards,

Chua Tianxiang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message