giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [giraph] dlogothetis commented on a change in pull request #110: Counter Mechanism
Date Tue, 05 Nov 2019 19:19:50 GMT
dlogothetis commented on a change in pull request #110: Counter Mechanism
URL: https://github.com/apache/giraph/pull/110#discussion_r342750159
 
 

 ##########
 File path: giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java
 ##########
 @@ -1236,6 +1235,76 @@ public void cleanup(FinishedSuperstepStats finishedSuperstepStats)
       LOG.error("cleanup: Got InterruptedException on notification " +
           "to master about cleanup", e);
     }
+  }
+
+  /**
+   * Method to send the counter values to the master using the zookeeper
+   * This is called after finishing all the supersteps
+   * @param allSuperstepsDone Whether the job has finished all supersteps
+   * This is needed to ensure the superstep number is the same for master and
+   * worker, when all supersteps are finished
+   *
+   */
+  public void storeCountersInZooKeeper(boolean allSuperstepsDone) {
+    Set<CustomCounter> additionalCounters =
+            CustomCounters.getCustomCounters();
+
+    JSONArray jsonCounters = new JSONArray();
+    Mapper.Context context = getContext();
+    Counter counter;
+    for (CustomCounter customCounter : additionalCounters) {
+      String groupName = customCounter.getGroupName();
+      String counterName = customCounter.getCounterName();
+      counter = context.getCounter(groupName, counterName);
+      customCounter.setValue(counter.getValue());
+      jsonCounters.put(Base64.encodeBytes(
+              WritableUtils.writeToByteArray(customCounter)));
+    }
+    // Add the Netty counters
+    Map<String, Set<String>> nettyCounters =
+            NettyClient.getCounterGroupsAndNames();
+    for (Map.Entry<String, Set<String>> entry : nettyCounters.entrySet()) {
+      String groupName = entry.getKey();
+      for (String counterName: entry.getValue()) {
+        CustomCounter customCounter = new CustomCounter(groupName, counterName,
+                CustomCounter.AGGREGATION.SUM);
+        counter = context.getCounter(groupName, counterName);
+        customCounter.setValue(counter.getValue());
+        jsonCounters.put(Base64.encodeBytes(
+                WritableUtils.writeToByteArray(customCounter)));
+      }
+    }
+    long superStep = getSuperstep() + (allSuperstepsDone ? 1 : 0);
+    String finishedWorkerPath =
+            getWorkerFinishedPath(
+                    getApplicationAttempt(), superStep, true) +
+                    "/" + workerInfo.getHostnameId();
+    LOG.info("worker finished path: " + finishedWorkerPath);
 
 Review comment:
   Related to the commend about having a separate method for getting the path for the counters,
let's change the log line too, to reflect we're writing counters.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message