storm-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kabh...@apache.org
Subject [1/2] storm git commit: [STORM-934] Add doc descriptions for the TOPOLOGY_ACKER_EXECUTORS
Date Mon, 13 Jul 2015 01:00:06 GMT
Repository: storm
Updated Branches:
  refs/heads/master 9906f85d5 -> 7bccc6d3d


[STORM-934] Add doc descriptions for the TOPOLOGY_ACKER_EXECUTORS


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/c27948df
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/c27948df
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/c27948df

Branch: refs/heads/master
Commit: c27948df35fd6ccbbe6fd19c3c2528c810c24699
Parents: 9906f85
Author: zhuol <zhuol@yahoo-inc.com>
Authored: Fri Jul 10 13:38:06 2015 -0500
Committer: Jungtaek Lim <kabhwan@gmail.com>
Committed: Mon Jul 13 09:51:18 2015 +0900

----------------------------------------------------------------------
 docs/documentation/Guaranteeing-message-processing.md           | 4 ++--
 .../documentation/Running-topologies-on-a-production-cluster.md | 4 ++--
 storm-core/src/jvm/backtype/storm/Config.java                   | 5 +++--
 3 files changed, 7 insertions(+), 6 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/storm/blob/c27948df/docs/documentation/Guaranteeing-message-processing.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Guaranteeing-message-processing.md b/docs/documentation/Guaranteeing-message-processing.md
index 4ecc620..9ade3fa 100644
--- a/docs/documentation/Guaranteeing-message-processing.md
+++ b/docs/documentation/Guaranteeing-message-processing.md
@@ -136,7 +136,7 @@ As always in software design, the answer is "it depends." Storm 0.7.0
introduced
 
 ### How does Storm implement reliability in an efficient way?
 
-A Storm topology has a set of special "acker" tasks that track the DAG of tuples for every
spout tuple. When an acker sees that a DAG is complete, it sends a message to the spout task
that created the spout tuple to ack the message. You can set the number of acker tasks for
a topology in the topology configuration using [Config.TOPOLOGY_ACKERS](/javadoc/apidocs/backtype/storm/Config.html#TOPOLOGY_ACKERS).
Storm defaults TOPOLOGY_ACKERS to one task -- you will need to increase this number for topologies
processing large amounts of messages.
+A Storm topology has a set of special "acker" tasks that track the DAG of tuples for every
spout tuple. When an acker sees that a DAG is complete, it sends a message to the spout task
that created the spout tuple to ack the message. You can set the number of acker executors
for a topology in the topology configuration using [Config.TOPOLOGY_ACKER_EXECUTORS](/javadoc/apidocs/backtype/storm/Config.html#TOPOLOGY_ACKER_EXECUTORS).
Storm defaults TOPOLOGY_ACKER_EXECUTORS to be equal to the number of workers configured in
the topology -- you will need to increase this number for topologies processing large amounts
of messages.
 
 The best way to understand Storm's reliability implementation is to look at the lifecycle
of tuples and tuple DAGs. When a tuple is created in a topology, whether in a spout or a bolt,
it is given a random 64 bit id. These ids are used by ackers to track the tuple DAG for every
spout tuple.
 
@@ -178,4 +178,4 @@ There are three ways to remove reliability. The first is to set Config.TOPOLOGY_
 
 The second way is to remove reliability on a message by message basis. You can turn off tracking
for an individual spout tuple by omitting a message id in the `SpoutOutputCollector.emit`
method.
 
-Finally, if you don't care if a particular subset of the tuples downstream in the topology
fail to be processed, you can emit them as unanchored tuples. Since they're not anchored to
any spout tuples, they won't cause any spout tuples to fail if they aren't acked.
\ No newline at end of file
+Finally, if you don't care if a particular subset of the tuples downstream in the topology
fail to be processed, you can emit them as unanchored tuples. Since they're not anchored to
any spout tuples, they won't cause any spout tuples to fail if they aren't acked.

http://git-wip-us.apache.org/repos/asf/storm/blob/c27948df/docs/documentation/Running-topologies-on-a-production-cluster.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Running-topologies-on-a-production-cluster.md b/docs/documentation/Running-topologies-on-a-production-cluster.md
index ebba452..b8cb892 100644
--- a/docs/documentation/Running-topologies-on-a-production-cluster.md
+++ b/docs/documentation/Running-topologies-on-a-production-cluster.md
@@ -50,7 +50,7 @@ You can find out how to configure your `storm` client to talk to a Storm
cluster
 There are a variety of configurations you can set per topology. A list of all the configurations
you can set can be found [here](/javadoc/apidocs/backtype/storm/Config.html). The ones prefixed
with "TOPOLOGY" can be overridden on a topology-specific basis (the other ones are cluster
configurations and cannot be overridden). Here are some common ones that are set for a topology:
 
 1. **Config.TOPOLOGY_WORKERS**: This sets the number of worker processes to use to execute
the topology. For example, if you set this to 25, there will be 25 Java processes across the
cluster executing all the tasks. If you had a combined 150 parallelism across all components
in the topology, each worker process will have 6 tasks running within it as threads.
-2. **Config.TOPOLOGY_ACKERS**: This sets the number of tasks that will track tuple trees
and detect when a spout tuple has been fully processed. Ackers are an integral part of Storm's
reliability model and you can read more about them on [Guaranteeing message processing](Guaranteeing-message-processing.html).
+2. **Config.TOPOLOGY_ACKER_EXECUTORS**: This sets the number of executors that will track
tuple trees and detect when a spout tuple has been fully processed. Ackers are an integral
part of Storm's reliability model and you can read more about them on [Guaranteeing message
processing](Guaranteeing-message-processing.html). By not setting this variable or setting
it as null, Storm will set the number of acker executors to be equal to the number of workers
configured for this topology. If this variable is set to 0, then Storm will immediately ack
tuples as soon as they come off the spout, effectively disabling reliability.
 3. **Config.TOPOLOGY_MAX_SPOUT_PENDING**: This sets the maximum number of spout tuples that
can be pending on a single spout task at once (pending means the tuple has not been acked
or failed yet). It is highly recommended you set this config to prevent queue explosion.
 4. **Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS**: This is the maximum amount of time a spout tuple
has to be fully completed before it is considered failed. This value defaults to 30 seconds,
which is sufficient for most topologies. See [Guaranteeing message processing](Guaranteeing-message-processing.html)
for more information on how Storm's reliability model works.
 5. **Config.TOPOLOGY_SERIALIZATIONS**: You can register more serializers to Storm using this
config so that you can use custom types within tuples.
@@ -74,4 +74,4 @@ To update a running topology, the only option currently is to kill the current
t
 
 The best place to monitor a topology is using the Storm UI. The Storm UI provides information
about errors happening in tasks and fine-grained stats on the throughput and latency performance
of each component of each running topology.
 
-You can also look at the worker logs on the cluster machines.
\ No newline at end of file
+You can also look at the worker logs on the cluster machines.

http://git-wip-us.apache.org/repos/asf/storm/blob/c27948df/storm-core/src/jvm/backtype/storm/Config.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/Config.java b/storm-core/src/jvm/backtype/storm/Config.java
index 063d446..f0ae8e7 100644
--- a/storm-core/src/jvm/backtype/storm/Config.java
+++ b/storm-core/src/jvm/backtype/storm/Config.java
@@ -1080,8 +1080,9 @@ public class Config extends HashMap<String, Object> {
     /**
      * How many executors to spawn for ackers.
      *
-     * <p>If this is set to 0, then Storm will immediately ack tuples as soon
-     * as they come off the spout, effectively disabling reliability.</p>
+     * <p>By not setting this variable or setting it as null, Storm will set the number
of acker executors
+     * to be equal to the number of workers configured for this topology. If this variable
is set to 0,
+     * then Storm will immediately ack tuples as soon as they come off the spout, effectively
disabling reliability.</p>
      */
     public static final String TOPOLOGY_ACKER_EXECUTORS = "topology.acker.executors";
     public static final Object TOPOLOGY_ACKER_EXECUTORS_SCHEMA = ConfigValidation.IntegerValidator;


Mime
View raw message