storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bobby Evans <ev...@yahoo-inc.com>
Subject Re: HDFS state and commits and only-once semantics
Date Mon, 07 Aug 2017 15:00:15 GMT
Yes if a batch fails the state may be written to one, but not the other.  At least not until
the batch completes successfully.


- Bobby


On Thursday, August 3, 2017, 3:36:51 PM CDT, Geoffrey Holmes <gholmes@pinsightmedia.com>
wrote:

#yiv7886372101 #yiv7886372101 -- _filtered #yiv7886372101 {font-family:Arial;panose-1:2 11
6 4 2 2 2 2 2 4;} _filtered #yiv7886372101 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv7886372101
{font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv7886372101 {panose-1:2
0 5 3 0 0 0 2 0 4;}#yiv7886372101 #yiv7886372101 p.yiv7886372101MsoNormal, #yiv7886372101
li.yiv7886372101MsoNormal, #yiv7886372101 div.yiv7886372101MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;}#yiv7886372101
a:link, #yiv7886372101 span.yiv7886372101MsoHyperlink {color:blue;text-decoration:underline;}#yiv7886372101
a:visited, #yiv7886372101 span.yiv7886372101MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv7886372101
p.yiv7886372101msonormal, #yiv7886372101 li.yiv7886372101msonormal, #yiv7886372101 div.yiv7886372101msonormal
{margin-right:0in;margin-left:0in;font-size:11.0pt;}#yiv7886372101 p.yiv7886372101msochpdefault,
#yiv7886372101 li.yiv7886372101msochpdefault, #yiv7886372101 div.yiv7886372101msochpdefault
{margin-right:0in;margin-left:0in;font-size:11.0pt;}#yiv7886372101 span.yiv7886372101msohyperlink
{}#yiv7886372101 span.yiv7886372101msohyperlinkfollowed {}#yiv7886372101 span.yiv7886372101emailstyle17
{}#yiv7886372101 span.yiv7886372101msoins {}#yiv7886372101 p.yiv7886372101msonormal1, #yiv7886372101
li.yiv7886372101msonormal1, #yiv7886372101 div.yiv7886372101msonormal1 {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv7886372101
span.yiv7886372101msohyperlink1 {color:#0563C1;text-decoration:underline;}#yiv7886372101 span.yiv7886372101msohyperlinkfollowed1
{color:#954F72;text-decoration:underline;}#yiv7886372101 span.yiv7886372101emailstyle171 {color:windowtext;font-weight:normal;font-style:normal;}#yiv7886372101
span.yiv7886372101msoins1 {color:teal;text-decoration:underline;}#yiv7886372101 p.yiv7886372101msochpdefault1,
#yiv7886372101 li.yiv7886372101msochpdefault1, #yiv7886372101 div.yiv7886372101msochpdefault1
{margin-right:0in;margin-left:0in;font-size:11.0pt;}#yiv7886372101 span.yiv7886372101EmailStyle29
{color:windowtext;font-weight:normal;font-style:normal;}#yiv7886372101 span.yiv7886372101msoIns
{text-decoration:underline;color:teal;}#yiv7886372101 .yiv7886372101MsoChpDefault {font-size:10.0pt;}
_filtered #yiv7886372101 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv7886372101 div.yiv7886372101WordSection1
{}#yiv7886372101 
Thanks. That helps answer my question. What if a batch fails? Could records in that batch
get written to disk by one HDFS state but not another?
 
  
 
From:Bobby Evans <evans@yahoo-inc.com>
Reply-To: "user@storm.apache.org" <user@storm.apache.org>
Date: Thursday, August 3, 2017 at 1:27 PM
To: "user@storm.apache.org" <user@storm.apache.org>
Subject: Re: HDFS state and commits and only-once semantics
 
  
 
Writing to a state in storm in not atomic.  Storm guarantees that once the batch completes
that the data is written out to all of the states that expect to receive it.  The HDFS state
guarantees that the dat will have been flushed out the the data nodes when a batch completes
and if the topology keeps running eventually the files will be rotated and made available
for others to process, but there is no guarantee that the files will rotate at the same time
or anything like that.
 
  
 
  
 
- Bobby
 
  
 
  
 
  
 
On Thursday, August 3, 2017, 1:17:39 PM CDT, Geoffrey Holmes <gholmes@pinsightmedia.com>
wrote:
 
  
 
  
 
I read STORM-837 (https://issues.apache.org/jira/browse/STORM-837) and have a question. How
does this work if I have more than one HDFS state in my Trident topology? Can I ensure that
a record ends up written to both HDFS states or none but not just one or the other?
 
 
 
 
 
Mime
View raw message