storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillermo López <glo...@sentisis.com>
Subject Re: Some questions I have while evaluating Storm
Date Tue, 23 Dec 2014 21:42:23 GMT

El 23/12/14 a las 22:41, Guillermo López escribió:
>
> El 23/12/14 a las 22:17, Eyal Golan escribió:
>> quick question,
>> regarding 4 - how would you know that a bolt has received everything ?
> This is what I do:
>
> 1) On the prepare method of the bolt, I check the getThisSources of
> the TopologyContext [1], and save every input source
Oh, this is incorrect in the case you are listening to various streams
(not just the default one). In this case, you need to take care of
inputs based on the stream.
> 2) I use a RotatingMap to save the received tuples into a
> ArrayList<Tuple> (with the TupleID as a key).
> 3) When the number of tuples for a received tupleID equals the number
> of source inputs, you can ACK all tuples and accept that you have
> received an input from all the source components, and join the result
> as you want.
>
> This is, of course, for a topology that need to join the result of
> every other bolt that emits to the bolt.
>
> That's why someone asked you what is "everything" for you ;)
>
> Cheers,
>
> Guillermo
>
> [1]
> http://storm.apache.org/apidocs/backtype/storm/task/TopologyContext.html#getThisSources%28%29
> [2] http://storm.apache.org/apidocs/backtype/storm/utils/RotatingMap.html
>>
>> 10x
>>
>> /Eyal Golan/
>>
>> Connect
>>
>> Gmail <mailto:egolan74@gmail.com> LinkedIn
>> <http://www.linkedin.com/in/egolan74> My Blog
>> <http://eyalgo.com/> Twitter
>> <http://twitter.com/eyalgo_egolan> Skype GitHub
>> <https://github.com/eyalgo> Facebook
>> <http://www.facebook.com/eyal.golan.14>
>>
>>
>> P  Save a tree. Please don't print this e-mail unless it's really
>> necessary
>>
>> On Tue, Dec 23, 2014 at 8:49 PM, Nathan Leung <ncleung@gmail.com
>> <mailto:ncleung@gmail.com>> wrote:
>>
>>     1. You do not have to manually push to nimbus.  When you run
>>     "storm jar" it will automatically send everything that is needed
>>     to the nimbus using the thrift interface.
>>
>>     2. Nimbus manages this with the supervisors.
>>
>>     3. You would need to write a custom scheduler.  See for
>>     example http://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/
>>
>>     4. Yes, you would need to store the tuples in the bolt until you
>>     have received everything you expect, then emit the output tuple
>>     after the last tuple has arrived.
>>
>>     On Tue, Dec 23, 2014 at 12:31 PM, Tim Molter
>>     <tim.molter@gmail.com <mailto:tim.molter@gmail.com>> wrote:
>>
>>         I'm hoping someone with practical experience can answer some
>>         questions I
>>         have. I have already scoured the docs and watched some
>>         videos, but I
>>         still have some unanswered questions.
>>
>>         1. When deploying to a cluster, do I always have to build a
>>         new jar,
>>         manually push it to the Nimbus machine and run "storm jar my.jar
>>         Myclass" or can I run a jar locally that calls
>>         "StormSubmitter.submit"
>>         and everything is taken care of?
>>
>>         2. Does Nimbus then push jars with the new implementation
>>         code to all
>>         the workers or does that have to be manually handled?
>>
>>         3. Can you configure the cluster so that it only run certain
>>         bolts on
>>         certain machines? How?
>>
>>         4. Can you join tuple streams and only send output tuples
>>         downstream
>>         after all expected input tuples have been received?
>>
>>         Thanks in advance!
>>
>>         ~Tim
>>
>>
>>
>


Mime
View raw message