storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillermo López <glo...@sentisis.com>
Subject Re: Some questions I have while evaluating Storm
Date Tue, 23 Dec 2014 21:41:15 GMT

El 23/12/14 a las 22:17, Eyal Golan escribió:
> quick question,
> regarding 4 - how would you know that a bolt has received everything ?
This is what I do:

1) On the prepare method of the bolt, I check the getThisSources of the
TopologyContext [1], and save every input source
2) I use a RotatingMap to save the received tuples into a
ArrayList<Tuple> (with the TupleID as a key).
3) When the number of tuples for a received tupleID equals the number of
source inputs, you can ACK all tuples and accept that you have received
an input from all the source components, and join the result as you want.

This is, of course, for a topology that need to join the result of every
other bolt that emits to the bolt.

That's why someone asked you what is "everything" for you ;)

Cheers,

Guillermo

[1]
http://storm.apache.org/apidocs/backtype/storm/task/TopologyContext.html#getThisSources%28%29
[2] http://storm.apache.org/apidocs/backtype/storm/utils/RotatingMap.html
>
> 10x
>
> /Eyal Golan/
>
> Connect
>
> Gmail <mailto:egolan74@gmail.com> LinkedIn
> <http://www.linkedin.com/in/egolan74> My Blog
> <http://eyalgo.com/> Twitter
> <http://twitter.com/eyalgo_egolan> Skype GitHub
> <https://github.com/eyalgo> Facebook
> <http://www.facebook.com/eyal.golan.14>
>
>
> P  Save a tree. Please don't print this e-mail unless it's really
> necessary
>
> On Tue, Dec 23, 2014 at 8:49 PM, Nathan Leung <ncleung@gmail.com
> <mailto:ncleung@gmail.com>> wrote:
>
>     1. You do not have to manually push to nimbus.  When you run
>     "storm jar" it will automatically send everything that is needed
>     to the nimbus using the thrift interface.
>
>     2. Nimbus manages this with the supervisors.
>
>     3. You would need to write a custom scheduler.  See for
>     example http://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/
>
>     4. Yes, you would need to store the tuples in the bolt until you
>     have received everything you expect, then emit the output tuple
>     after the last tuple has arrived.
>
>     On Tue, Dec 23, 2014 at 12:31 PM, Tim Molter <tim.molter@gmail.com
>     <mailto:tim.molter@gmail.com>> wrote:
>
>         I'm hoping someone with practical experience can answer some
>         questions I
>         have. I have already scoured the docs and watched some videos,
>         but I
>         still have some unanswered questions.
>
>         1. When deploying to a cluster, do I always have to build a
>         new jar,
>         manually push it to the Nimbus machine and run "storm jar my.jar
>         Myclass" or can I run a jar locally that calls
>         "StormSubmitter.submit"
>         and everything is taken care of?
>
>         2. Does Nimbus then push jars with the new implementation code
>         to all
>         the workers or does that have to be manually handled?
>
>         3. Can you configure the cluster so that it only run certain
>         bolts on
>         certain machines? How?
>
>         4. Can you join tuple streams and only send output tuples
>         downstream
>         after all expected input tuples have been received?
>
>         Thanks in advance!
>
>         ~Tim
>
>
>


Mime
View raw message