hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11160) Auto-gather column stats
Date Fri, 08 Dec 2017 20:55:01 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284189#comment-16284189
] 

Hive QA commented on HIVE-11160:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12799200/HIVE-11160.09.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8148/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8148/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8148/

Messages:
{noformat}
**** This message was trimmed, see log for full details ****
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8148/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-12-08 20:52:55.963
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 36f0d89 HIVE-17333 Oracle does not allow change from VARCHAR2 to CLOB for upgrade
(Naveen Gangam, reviewed by Aihua Xu)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 36f0d89 HIVE-17333 Oracle does not allow change from VARCHAR2 to CLOB for upgrade
(Naveen Gangam, reviewed by Aihua Xu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-12-08 20:53:00.664
+ rm -rf ../yetus
+ mkdir ../yetus
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8148/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: No such file or directory
error: a/data/conf/hive-site.xml: No such file or directory
error: a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
No such file or directory
error: a/itests/src/test/resources/testconfiguration.properties: No such file or directory
error: a/metastore/if/hive_metastore.thrift: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp: No such
file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h: No such file or directory
error: a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SetPartitionsStatsRequest.java:
No such file or directory
error: a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:
No such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php: No such file
or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/Types.php: No such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote: No such
file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py: No such file
or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py: No such file or directory
error: a/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb: No such file or directory
error: a/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: No such file
or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java: No
such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: No such
file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java: No such
file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java: No such
file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java:
No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java:
No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java:
No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java:
No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java:
No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java:
No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/IExtrapolatePartStatus.java:
No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java:
No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java:
No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java: No such file or
directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java: No such
file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java: No
such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java:
No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java: No such
file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java: No such file or
directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java: No
such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: No such file or
directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java: No such file or
directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java: No such
file or directory
error: a/ql/src/test/queries/clientpositive/combine1.q: No such file or directory
error: a/ql/src/test/results/clientpositive/alter_merge_stats_orc.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table2_h23.q.out:
No such file or directory
error: a/ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table_h23.q.out:
No such file or directory
error: a/ql/src/test/results/clientpositive/alter_partition_coltype.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/alter_table_serde2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/analyze_table_null_partition.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/annotate_stats_filter.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/annotate_stats_groupby.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/auto_join1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join14.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join17.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join19.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join19_inclause.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join25.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join26.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join4.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join5.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join6.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join7.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join8.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join9.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join_reordering_values.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_sortmerge_join_13.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/binary_output_format.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin13.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin4.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin5.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_1.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_3.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_8.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/case_sensitivity.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/cast1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_1.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/columnstats_part_coltype.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/combine2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/constprog_dp.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/constprog_type.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/correlationoptimizer5.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/cp_sel.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/dynamic_partition_skip_default.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/explain_ddl.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/extrapolate_part_stats_full.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/extrapolate_part_stats_partial.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/extrapolate_part_stats_partial_ndv.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/filter_join_breaktask.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby10.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby11.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby12.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1_limit.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1_map_nomap.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1_noskew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby2_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby2_map_multi_distinct.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/groupby2_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby2_noskew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby2_noskew_multi_distinct.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/groupby3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby3_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby3_map_multi_distinct.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/groupby3_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby3_noskew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby3_noskew_multi_distinct.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/groupby4.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby4_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby4_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby4_noskew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby5_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby5_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby6.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby6_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby6_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby6_noskew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_cube_multi_gby.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/groupby_map_ppr.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_map_ppr_multi_distinct.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/groupby_multi_insert_common_distinct.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/groupby_multi_single_reducer.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/groupby_multi_single_reducer2.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/groupby_ppr.q.out: No such file or directory
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12799200 - PreCommit-HIVE-Build

> Auto-gather column stats
> ------------------------
>
>                 Key: HIVE-11160
>                 URL: https://issues.apache.org/jira/browse/HIVE-11160
>             Project: Hive
>          Issue Type: New Feature
>            Reporter: Pengcheng Xiong
>            Assignee: Pengcheng Xiong
>         Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, HIVE-11160.03.patch, HIVE-11160.04.patch,
HIVE-11160.05.patch, HIVE-11160.06.patch, HIVE-11160.07.patch, HIVE-11160.08.patch, HIVE-11160.09.patch
>
>
>  Hive will collect table stats when set hive.stats.autogather=true during the INSERT
OVERWRITE command. And then the users need to collect the column stats themselves using "Analyze"
command. In this patch, the column stats will also be collected automatically. More specifically,
INSERT OVERWRITE will automatically create new column stats. INSERT INTO will automatically
merge new column stats with existing ones.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message