storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "J.R. Pauley" <>
Subject Re: HDFS state and commits and only-once semantics
Date Thu, 03 Aug 2017 20:30:23 GMT
Yes, thanks - but I was inadvertently explicitly emitting both "args", and
"A,B" in my Fn's which was throwing me off. Got past that bit.

Seems that it is darn hard what I want to do insofar as the A,B,C params I
was emitting are just meant for internal consumption, not wanting them
present in the output stream. So my final output (http response which goes
back to drpc client) has http results I want to pass back plus "args,A,B,C"
that I was only emitting from Fn to Fn for internal use. I don't see way to
edit that output stream either. If I filter out tuple containing "args" it
gets rid of everything

On Thu, Aug 3, 2017 at 2:26 PM, Bobby Evans <> wrote:

> Writing to a state in storm in not atomic.  Storm guarantees that once the
> batch completes that the data is written out to all of the states that
> expect to receive it.  The HDFS state guarantees that the dat will have
> been flushed out the the data nodes when a batch completes and if the
> topology keeps running eventually the files will be rotated and made
> available for others to process, but there is no guarantee that the files
> will rotate at the same time or anything like that.
> - Bobby
> On Thursday, August 3, 2017, 1:17:39 PM CDT, Geoffrey Holmes <
>> wrote:
> I read STORM-837 ( and
> have a question. How does this work if I have more than one HDFS state in
> my Trident topology? Can I ensure that a record ends up written to both
> HDFS states or none but not just one or the other?

View raw message