hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-24946) Handle failover case during Repl Load
Date Thu, 29 Jul 2021 01:12:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-24946?focusedWorklogId=630870&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-630870
]

ASF GitHub Bot logged work on HIVE-24946:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 29/Jul/21 01:11
            Start Date: 29/Jul/21 01:11
    Worklog Time Spent: 10m 
      Work Description: hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678751529



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -554,6 +555,29 @@ public void run() throws SemanticException {
           }
         }
       });
+      if (work.shouldFailover()) {
+        listOfPreAckTasks.add(new PreAckTask() {
+          @Override
+          public void run() throws SemanticException {
+            try {
+              Database db = getHive().getDatabase(work.getTargetDatabase());
+              Map<String, String> params = db.getParameters();
+              if (params == null) {
+                params = new HashMap<>();
+                db.setParameters(params);
+              } else if (MetaStoreUtils.isTargetOfReplication(db)) {
+                params.remove(ReplConst.TARGET_OF_REPLICATION);

Review comment:
       In rollback, this prop would be again set in incremental load.

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadWork.java
##########
@@ -129,16 +130,20 @@ public ReplLoadWork(HiveConf hiveConf, String dumpDirectory,
       if (metricCollector != null) {
         metricCollector.setMetricsMBean(name);
       }
+      Path failoverReadyMarker = new Path(dumpDirectory, ReplAck.FAILOVER_READY_MARKER.toString());
+      FileSystem fs = failoverReadyMarker.getFileSystem(hiveConf);
+      shouldFailover = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_REPL_FAILOVER_START)
+              && fs.exists(failoverReadyMarker);
       incrementalLoadTasksBuilder = new IncrementalLoadTasksBuilder(dbNameToLoadIn, dumpDirectory,
           new IncrementalLoadEventsIterator(dumpDirectory, hiveConf), hiveConf, eventTo,
metricCollector,
-          replStatsTracker);
+          replStatsTracker, shouldFailover);
 
       /*
        * If the current incremental dump also includes bootstrap for some tables, then create
iterator
        * for the same.
        */
       Path incBootstrapDir = new Path(dumpDirectory, ReplUtils.INC_BOOTSTRAP_ROOT_DIR_NAME);
-      FileSystem fs = incBootstrapDir.getFileSystem(hiveConf);
+      fs = incBootstrapDir.getFileSystem(hiveConf);

Review comment:
       Removed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 630870)
    Time Spent: 1.5h  (was: 1h 20m)

> Handle failover case during Repl Load
> -------------------------------------
>
>                 Key: HIVE-24946
>                 URL: https://issues.apache.org/jira/browse/HIVE-24946
>             Project: Hive
>          Issue Type: New Feature
>            Reporter: Haymant Mangla
>            Assignee: Haymant Mangla
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> * Update metric during load to capture the readiness for failover
>  * Remove repl.target.for property on target cluster
>  * Prepare the dump directory to be used during failover first dump operation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message