storm-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bo...@apache.org
Subject [10/50] [abbrv] storm git commit: STORM-166: modifed some config key names to match the convention, added the documentation for configuration.
Date Mon, 24 Aug 2015 13:51:55 GMT
STORM-166: modifed some config key names to match the convention, added the documentation for
configuration.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/27d6b4ca
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/27d6b4ca
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/27d6b4ca

Branch: refs/heads/master
Commit: 27d6b4ca5ab4d4f9d7753d66b2a28c982b47c7e7
Parents: a75c72c
Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
Authored: Fri Dec 19 14:31:49 2014 -0800
Committer: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
Committed: Fri Dec 19 14:31:49 2014 -0800

----------------------------------------------------------------------
 conf/defaults.yaml                              |  4 ++--
 docs/documentation/nimbus-ha-design.md          | 23 ++++++++++++++++++++
 .../src/clj/backtype/storm/daemon/nimbus.clj    |  4 ++--
 storm-core/src/jvm/backtype/storm/Config.java   | 13 +++++------
 4 files changed, 33 insertions(+), 11 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/storm/blob/27d6b4ca/conf/defaults.yaml
----------------------------------------------------------------------
diff --git a/conf/defaults.yaml b/conf/defaults.yaml
index 07230dc..e189966 100644
--- a/conf/defaults.yaml
+++ b/conf/defaults.yaml
@@ -70,8 +70,8 @@ nimbus.task.launch.secs: 120
 nimbus.reassign: true
 nimbus.file.copy.expiration.secs: 600
 nimbus.topology.validator: "backtype.storm.nimbus.DefaultTopologyValidator"
-min.replication.count: 0
-max.replication.wait.time.sec: 0
+nimbus.min.replication.count: 0
+nimbus.max.replication.wait.time.sec: 0
 nimbus.credential.renewers.freq.secs: 600
 
 ### ui.* configs are for the master

http://git-wip-us.apache.org/repos/asf/storm/blob/27d6b4ca/docs/documentation/nimbus-ha-design.md
----------------------------------------------------------------------
diff --git a/docs/documentation/nimbus-ha-design.md b/docs/documentation/nimbus-ha-design.md
index cb6332f..9c38851 100644
--- a/docs/documentation/nimbus-ha-design.md
+++ b/docs/documentation/nimbus-ha-design.md
@@ -198,3 +198,26 @@ nimbus hosts. Any nimbus host will be able to respond to these requests.
The nim
 from zookeeper and cache it and keep updating the cache when the watchers are fired to indicate
any changes,which should be 
 rare in general case. In addition we should update all the existing thrift and rest apis’s
to throw redirect 
 exceptions when a non leader receives a request that only a leader should serve.
+
+## Configuration
+You can use nimbus ha with default configuration , however the default configuration assumes
a single nimbus host so it
+trades off replication for lower topology submission latency. Depending on your use case
you can adjust following configurations:
+* storm.codedistributor.class : This is a string representing fully qualified class name
of a class that implements
+backtype.storm.codedistributor.ICodeDistributor. The default is set to "backtype.storm.codedistributor.LocalFileSystemCodeDistributor".
+This class leverages local file system to store both meta files and code/configs. This class
adds extra load on zookeeper as even after
+downloading the code-distrbutor meta file it contacts zookeeper in order to figure out hosts
from where it can download
+actual code/config and to get the current replication count. An alternative is to use 
+"org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor" which relies on HDFS but does
not add extra load on zookeeper and will 
+make topology submission faster.
+* nimbus.min.replication.count : Minimum number of nimbus hosts where the code must be replicated
before leader nimbus
+can mark the topology as active and create assignments. Default is 0. in case of HDFSCodeDistributor
this represents number
+of data nodes insted of nimbus hosts where code must be replicated before activating topology.
+* nimbus.max.replication.wait.time.sec: Maximum wait time for the nimbus host replication
to achieve the nimbus.min.replication.count.
+Once this time is elapsed nimbus will go ahead and perform topology activation tasks even
if required nimbus.min.replication.count is not achieved. 
+The default is 0 seconds, a value of -1 indicates to wait for ever.
+*nimbus.code.sync.freq.secs: frequency at which the background thread which syncs code for
locally missing topologies will run. default is 5 minutes.
+
+Note: Even though all nimbus hosts have watchers on zookeeper to be notified immediately
as soon as a new topology is available for code
+download, due to eventual consistency of zookeeper the callback pretty much never results
in code download. In practice we have observed that
+the desired replication is only achieved once the background-thread runs. So you should expect
your topology submission time to be somewhere between
+0 to (2 * nimbus.code.sync.freq.secs) for any nimbus.min.replication.count > 0.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/storm/blob/27d6b4ca/storm-core/src/clj/backtype/storm/daemon/nimbus.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/nimbus.clj b/storm-core/src/clj/backtype/storm/daemon/nimbus.clj
index b638e52..0281e73 100644
--- a/storm-core/src/clj/backtype/storm/daemon/nimbus.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/nimbus.clj
@@ -340,8 +340,8 @@
    ))
 
 (defn- wait-for-desired-code-replication [nimbus conf storm-id]
-  (let [min-replication-count (conf MIN-REPLICATION-COUNT)
-        max-replication-wait-time (conf MAX-REPLICATION-WAIT-TIME-SEC)
+  (let [min-replication-count (conf NIMBUS-MIN-REPLICATION-COUNT)
+        max-replication-wait-time (conf NIMBUS-MAX-REPLICATION-WAIT-TIME-SEC)
         total-wait-time (atom 0)
         current-replication-count (atom (if (:bt-tracker nimbus) (.getReplicationCount (:bt-tracker
nimbus) storm-id) 0))]
   (if (:bt-tracker nimbus)

http://git-wip-us.apache.org/repos/asf/storm/blob/27d6b4ca/storm-core/src/jvm/backtype/storm/Config.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/Config.java b/storm-core/src/jvm/backtype/storm/Config.java
index ead144f..9746565 100644
--- a/storm-core/src/jvm/backtype/storm/Config.java
+++ b/storm-core/src/jvm/backtype/storm/Config.java
@@ -1288,18 +1288,17 @@ public class Config extends HashMap<String, Object> {
      * is allowed to perform topology activation tasks like setting up heartbeats/assignments
      * and marking the topology as active. default is 0.
      */
-    public static final String MIN_REPLICATION_COUNT = "min.replication.count";
-    public static final Object MIN_REPLICATION_COUNT_SCHEMA = Number.class;
+    public static final String NIMBUS_MIN_REPLICATION_COUNT = "nimbus.min.replication.count";
+    public static final Object NIMBUS_MIN_REPLICATION_COUNT_SCHEMA = Number.class;
 
     /**
-     * Maximum wait time for the nimbus host replication to achieve the min.replication.count.
+     * Maximum wait time for the nimbus host replication to achieve the nimbus.min.replication.count.
      * Once this time is elapsed nimbus will go ahead and perform topology activation tasks
even
-     * if required min.replication.count is not achieved. The default is 0 seconds, a value
of
+     * if required nimbus.min.replication.count is not achieved. The default is 0 seconds,
a value of
      * -1 indicates to wait for ever.
      */
-    public static final String MAX_REPLICATION_WAIT_TIME_SEC = "max.replication.wait.time.sec";
-    public static final Object MAX_REPLICATION_WAIT_TIME_SEC_SCHEMA = Number.class;
-
+    public static final String NIMBUS_MAX_REPLICATION_WAIT_TIME_SEC = "nimbus.max.replication.wait.time.sec";
+    public static final Object NIMBUS_MAX_REPLICATION_WAIT_TIME_SEC_SCHEMA = Number.class;
 
     /**
      * How often nimbus should wake the cleanup thread to clean the inbox.


Mime
View raw message