storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bobby Evans <>
Subject Re: HDFS state and commits and only-once semantics
Date Thu, 03 Aug 2017 18:26:50 GMT
Writing to a state in storm in not atomic.  Storm guarantees that once the batch completes
that the data is written out to all of the states that expect to receive it.  The HDFS state
guarantees that the dat will have been flushed out the the data nodes when a batch completes
and if the topology keeps running eventually the files will be rotated and made available
for others to process, but there is no guarantee that the files will rotate at the same time
or anything like that.

- Bobby

On Thursday, August 3, 2017, 1:17:39 PM CDT, Geoffrey Holmes <>

<!--#yiv9469310365 _filtered #yiv9469310365 {font-family:Arial;panose-1:2 11 6 4 2 2 2
2 2 4;} _filtered #yiv9469310365 {font-family:"Cambria Math";panose-1:2 4 5 3 5 4 6 3 2 4;}
_filtered #yiv9469310365 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv9469310365
#yiv9469310365 p.yiv9469310365MsoNormal, #yiv9469310365 li.yiv9469310365MsoNormal, #yiv9469310365
div.yiv9469310365MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;font-family:"Calibri",
sans-serif;}#yiv9469310365 a:link, #yiv9469310365 span.yiv9469310365MsoHyperlink {color:#0563C1;text-decoration:underline;}#yiv9469310365
a:visited, #yiv9469310365 span.yiv9469310365MsoHyperlinkFollowed {color:#954F72;text-decoration:underline;}#yiv9469310365
span.yiv9469310365EmailStyle17 {font-family:"Arial", sans-serif;color:windowtext;font-weight:normal;font-style:normal;}#yiv9469310365
span.yiv9469310365msoIns {text-decoration:underline;color:teal;}#yiv9469310365 .yiv9469310365MsoChpDefault
{font-family:"Calibri", sans-serif;} _filtered #yiv9469310365 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv9469310365
div.yiv9469310365WordSection1 {}-->
I read STORM-837 ( and have a question. How
does this work if I have more than one HDFS state in my Trident topology? Can I ensure that
a record ends up written to both HDFS states or none but not just one or the other?
View raw message