storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navin Ipe <navin....@searchlighthealth.com>
Subject How to let a topology know that it's time to stop?
Date Thu, 05 May 2016 04:13:37 GMT
Hi,

I know Storm is designed to run forever. I also know about Trident's
technique of aggregation. But shouldn't Storm have a way to let bolts know
that a certain bunch of processing has been completed?

Consider this topology:
Spout------>Bolt-A------>Bolt-B
            |                  |--->Bolt-B
            |                  \--->Bolt-B
            |--->Bolt-A------>Bolt-B
            |                  |--->Bolt-B
            |                  \--->Bolt-B
            \--->Bolt-A------>Bolt-B
                               |--->Bolt-B
                               \--->Bolt-B


   - From Bolt-A to Bolt-B, it is a FieldsGrouping.
   - Spout emits only a few tuples and then stops emitting.
   - Bolt A takes those tuples and generates millions of tuples.


*Bolt-B accumulates tuples that Bolt A sends and needs to know when Spout
finished emitting. Only then can Bolt-B start writing to SQL.*

*Questions:*
1. How can all Bolts B be notified that it is time to write to SQL?
2. After all Bolts B have written to SQL, how to know that all Bolts B have
completed writing?
3. How to stop the topology? I know of
localCluster.killTopology("HelloStorm"), but shouldn't there be a way to do
it from the Bolt?

-- 
Regards,
Navin

Mime
View raw message