hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-25373) Modify buildColumnStatsDesc to send configured number of stats for updation
Date Wed, 28 Jul 2021 04:19:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-25373?focusedWorklogId=628861&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628861
]

ASF GitHub Bot logged work on HIVE-25373:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 28/Jul/21 04:18
            Start Date: 28/Jul/21 04:18
    Worklog Time Spent: 10m 
      Work Description: rbalamohan commented on a change in pull request #2520:
URL: https://github.com/apache/hive/pull/2520#discussion_r677960313



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java
##########
@@ -177,25 +182,36 @@ private ColumnStatisticsDesc buildColumnStatsDesc(Table table, String
partName,
   public int persistColumnStats(Hive db, Table tbl) throws HiveException, MetaException,
IOException {
     // Construct a column statistics object from the result
 
-    List<ColumnStatistics> colStats = constructColumnStatsFromPackedRows(tbl);
-    // Persist the column statistics object to the metastore
-    // Note, this function is shared for both table and partition column stats.
-    if (colStats.isEmpty()) {
-      return 0;
-    }
-    SetPartitionsStatsRequest request = new SetPartitionsStatsRequest(colStats, Constants.HIVE_ENGINE);
-    request.setNeedMerge(colStatDesc.isNeedMerge());
-    HiveTxnManager txnMgr = AcidUtils.isTransactionalTable(tbl)
-        ? SessionState.get().getTxnMgr() : null;
+    long writeId = -1;
+    ValidWriteIdList validWriteIdList = null;
+    HiveTxnManager txnMgr = AcidUtils.isTransactionalTable(tbl) ? SessionState.get().getTxnMgr()
: null;
     if (txnMgr != null) {
-      request.setWriteId(txnMgr.getAllocatedTableWriteId(tbl.getDbName(), tbl.getTableName()));
-      ValidWriteIdList validWriteIdList =
-          AcidUtils.getTableValidWriteIdList(conf, AcidUtils.getFullTableName(tbl.getDbName(),
tbl.getTableName()));
-      if (validWriteIdList != null) {
-        request.setValidWriteIdList(validWriteIdList.toString());
+      writeId = txnMgr.getAllocatedTableWriteId(tbl.getDbName(), tbl.getTableName());
+      validWriteIdList =
+              AcidUtils.getTableValidWriteIdList(conf, AcidUtils.getFullTableName(tbl.getDbName(),
tbl.getTableName()));
+    }
+
+    boolean done = false;
+    long maxNumStats = conf.getLongVar(HiveConf.ConfVars.HIVE_STATS_MAX_NUM_STATS);
+    while (!done) {
+      List<ColumnStatistics> colStats = new ArrayList<>();
+      done = constructColumnStatsFromPackedRows(tbl, colStats, maxNumStats);
+
+      // Persist the column statistics object to the metastore
+      // Note, this function is shared for both table and partition column stats.
+      if (colStats.isEmpty()) {
+        continue;
+      }
+      SetPartitionsStatsRequest request = new SetPartitionsStatsRequest(colStats, Constants.HIVE_ENGINE);
+      request.setNeedMerge(colStatDesc.isNeedMerge());
+      if (txnMgr != null) {
+        request.setWriteId(writeId);
+        if (validWriteIdList != null) {
+          request.setValidWriteIdList(validWriteIdList.toString());
+        }
       }
+      db.setPartitionColumnStatistics(request);

Review comment:
       Info level logging would be good to have here. Since it will be printed for every 10K
updates, it should be fine and will be helpful for debugging as well. And if possible provide
time taken for overall buildColumnStatsDesc.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 628861)
    Time Spent: 20m  (was: 10m)

> Modify buildColumnStatsDesc to send configured number of stats for updation
> ---------------------------------------------------------------------------
>
>                 Key: HIVE-25373
>                 URL: https://issues.apache.org/jira/browse/HIVE-25373
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: mahesh kumar behera
>            Assignee: mahesh kumar behera
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> The number of stats sent for updation should be controlled to avoid thrift error in case
the size exceeds the limit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message