storm-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [08/12] storm git commit: updated
Date Fri, 14 Aug 2015 18:01:24 GMT


Branch: refs/heads/master
Commit: 4d8c51b4503110f03a27b401f096ff2477bbb7cf
Parents: cdabbb3
Author: Arun Mahadevan <>
Authored: Mon Aug 10 14:01:24 2015 +0530
Committer: Arun Mahadevan <>
Committed: Mon Aug 10 14:01:24 2015 +0530

 external/storm-hdfs/ | 3 +++
 1 file changed, 3 insertions(+)
diff --git a/external/storm-hdfs/ b/external/storm-hdfs/
index 98fcc55..e1f2b28 100644
--- a/external/storm-hdfs/
+++ b/external/storm-hdfs/
@@ -323,6 +323,9 @@ duplicates from the current data file by copying the data up to the last
 operation involves a lot of data copy, ensure that the data files are rotated at reasonable
sizes with `FileSizeRotationPolicy` 
 and at reasonable intervals with `TimedRotationPolicy` so that the recovery can complete
within topology.message.timeout.secs.
+Also note with `TimedRotationPolicy` the files are never rotated in the middle of a batch
even if the timer ticks, 
+but only when a batch completes so that complete batches can be efficiently recovered in
case of failures.
 ##Working with Secure HDFS
 If your topology is going to interact with secure HDFS, your bolts/states needs to be authenticated
by NameNode. We 
 currently have 2 options to support this:

View raw message