storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eyal Golan <egola...@gmail.com>
Subject Re: Shared static data between bolts
Date Tue, 09 Dec 2014 08:37:59 GMT
Quick update for future readers :)
The shared collection, using static instances works.

>From this task I learned:
- Instantiate in the 'prepare' method and not in constructor
- Don't pass shared objects from the main to the bolts
- Good (?) old fashioned singleton pattern with synchronized

Eyal



*Eyal Golan*

Connect

[image: Gmail] <egolan74@gmail.com> [image: LinkedIn]
<http://www.linkedin.com/in/egolan74> [image: My Blog]
<http://eyalgo.com/> [image:
Twitter] <http://twitter.com/eyalgo_egolan> [image: Skype] [image: GitHub]
<https://github.com/eyalgo> [image: Facebook]
<http://www.facebook.com/eyal.golan.14>


P  Save a tree. Please don't print this e-mail unless it's really necessary

On Thu, Dec 4, 2014 at 11:20 PM, Dima Dragan <dima.dragan@belleron.net>
wrote:

> Hi Eyal,
>
> I used the same approach as Michael.
>
> As far as I understand, your shared collection should be changeable.  So
> do not forget use concurrent collection.
>
> Solution for copies by worker are:
>
> 1) Represent one collection as group of collections (collection per
> worker), using storm grouping make sure that input bolts emit to the
> appropriate cache-bolts.
>
> 2) use external storage for state persistence.
>
> It depends from your needs.
>
> Best regards,
> Dmytro Dragan
> On Dec 4, 2014 10:54 PM, "Michael Rose" <michael@fullcontact.com> wrote:
>
>> Yes, as long as they're in the same JVM. If you have 4 workers, that's 4
>> JVMs, and 4 copies of the data.
>>
>> Storm will ensure that if you have 4 bolts of type A, and 8 of type B,
>> that 1 of type A and 2 of type B exist in each JVM. Since the singleton's
>> field is static, it'll be shared.
>>
>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>> michael@fullcontact.com
>>
>> On Thu, Dec 4, 2014 at 1:34 PM, Eyal Golan <egolan74@gmail.com> wrote:
>>
>>> Michael,
>>> I understand your solution, however, if bolt of type A "wins" the
>>> initialization, will bolts of type B see the same instance?
>>>
>>> Sent from my Android phone
>>> On Dec 4, 2014 10:26 PM, "Eyal Golan" <egolan74@gmail.com> wrote:
>>>
>>>> Thanks Michael and Ben !
>>>>
>>>> Sent from my Android phone
>>>> On Dec 4, 2014 9:53 PM, "Flint, Ben" <bflint@popcap.com> wrote:
>>>>
>>>>>  I’ve used Hazelcast for use cases like this with great success.  It
>>>>> makes it relatively straightforward to implement a distributed data
>>>>> structure.
>>>>>
>>>>>   From: Eyal Golan <egolan74@gmail.com>
>>>>> Reply-To: "user@storm.apache.org" <user@storm.apache.org>
>>>>> Date: Thursday, December 4, 2014 at 11:40 AM
>>>>> To: "user@storm.apache.org" <user@storm.apache.org>
>>>>> Subject: Re: Shared static data between bolts
>>>>>
>>>>>   Thanks!
>>>>> All the answers give me good idea.
>>>>> Here's why I need this list be sharable.
>>>>> One bolt needs the list for computation.
>>>>> Another bolt will periodically update the list.
>>>>> In other words, the list is changeable.
>>>>>
>>>>> I tried passing it in conf but there are some 3rd party libs in the
>>>>> wrapper.
>>>>>
>>>>> Anyway, I also need to improve the topology. Separate bolt to two etc.
>>>>> That way, it would be easier to handle that cache.
>>>>>
>>>>> One more question, I thought about giving hazelcast a try.
>>>>> Create local node and then the bolts will have clients.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Sent from my Android phone
>>>>> On Dec 4, 2014 9:14 PM, "Kushan Maskey" <
>>>>> kushan.maskey@mmillerassociates.com> wrote:
>>>>>
>>>>>>  Hi Eyal,
>>>>>>
>>>>>>  I had the similar issue earlier. Static objects are not a good way
>>>>>> of passing the variables around in Storm due to clustered nature
of the
>>>>>> Storm. You will see null for all the static variables.
>>>>>>
>>>>>>  This is how I implemented mine, for instance I wanted to pass
>>>>>> around the server properties such and host name and username/password.
I
>>>>>> read the properties file and then created a map of these properties
then
>>>>>> set it into the Config object. Like below.
>>>>>>
>>>>>>  *In topology:*
>>>>>>
>>>>>>  Map<Stirng, Stirng> map = new HashMap();
>>>>>> map.put("hostname", "localhost");
>>>>>>  map.put("username", "testuser");
>>>>>>  map.put("password", "testpassord");
>>>>>> map.put("port", "1234");
>>>>>>
>>>>>> Config config = new Config();
>>>>>>
>>>>>> config.put("MY_MAP", map)
>>>>>>  *In Bolts:*
>>>>>>
>>>>>> public void prepare(Map stormConf, TopologyContext context) {
>>>>>>
>>>>>> HashMap<Stirng, Stirng> map = stormConfig.get("MY_MAP");
>>>>>>
>>>>>> }
>>>>>>  This how i resolved my problem.Try that if that helps.
>>>>>>   --
>>>>>> Kushan Maskey
>>>>>> 817.403.7500
>>>>>> M. Miller & Associates <http://mmillerassociates.com/>
>>>>>> kushan.maskey@mmillerassociates.com
>>>>>>
>>>>>> On Thu, Dec 4, 2014 at 1:01 PM, Eyal Golan <egolan74@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>> I am newbie with Storm and having some issues with static objects
in
>>>>>>> the topology.
>>>>>>> In the main class, where I set the topology, I also set a static
>>>>>>> object. It's actually a wrapper for a collection of strings.
>>>>>>> This object needs to he shared between different bolts.
>>>>>>> It's big so I want it to be only one instance.
>>>>>>>
>>>>>>> The problem is that when a bolt tries to get this static instance:
>>>>>>> example - MyMain.Shared... It's null. Although I instantiated
it in the
>>>>>>> main method.
>>>>>>>
>>>>>>> I guess I'm doing something wrong.
>>>>>>>
>>>>>>> Thanks for any assistance.
>>>>>>>
>>>>>>> Eyal
>>>>>>>
>>>>>>> Sent from my Android phone
>>>>>>>
>>>>>>
>>>>>>
>>

Mime
View raw message