storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dima Dragan <dima.dra...@belleron.net>
Subject Re: Shared static data between bolts
Date Thu, 04 Dec 2014 21:20:22 GMT
Hi Eyal,

I used the same approach as Michael.

As far as I understand, your shared collection should be changeable.  So do
not forget use concurrent collection.

Solution for copies by worker are:

1) Represent one collection as group of collections (collection per
worker), using storm grouping make sure that input bolts emit to the
appropriate cache-bolts.

2) use external storage for state persistence.

It depends from your needs.

Best regards,
Dmytro Dragan
On Dec 4, 2014 10:54 PM, "Michael Rose" <michael@fullcontact.com> wrote:

> Yes, as long as they're in the same JVM. If you have 4 workers, that's 4
> JVMs, and 4 copies of the data.
>
> Storm will ensure that if you have 4 bolts of type A, and 8 of type B,
> that 1 of type A and 2 of type B exist in each JVM. Since the singleton's
> field is static, it'll be shared.
>
> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
> michael@fullcontact.com
>
> On Thu, Dec 4, 2014 at 1:34 PM, Eyal Golan <egolan74@gmail.com> wrote:
>
>> Michael,
>> I understand your solution, however, if bolt of type A "wins" the
>> initialization, will bolts of type B see the same instance?
>>
>> Sent from my Android phone
>> On Dec 4, 2014 10:26 PM, "Eyal Golan" <egolan74@gmail.com> wrote:
>>
>>> Thanks Michael and Ben !
>>>
>>> Sent from my Android phone
>>> On Dec 4, 2014 9:53 PM, "Flint, Ben" <bflint@popcap.com> wrote:
>>>
>>>>  I’ve used Hazelcast for use cases like this with great success.  It
>>>> makes it relatively straightforward to implement a distributed data
>>>> structure.
>>>>
>>>>   From: Eyal Golan <egolan74@gmail.com>
>>>> Reply-To: "user@storm.apache.org" <user@storm.apache.org>
>>>> Date: Thursday, December 4, 2014 at 11:40 AM
>>>> To: "user@storm.apache.org" <user@storm.apache.org>
>>>> Subject: Re: Shared static data between bolts
>>>>
>>>>   Thanks!
>>>> All the answers give me good idea.
>>>> Here's why I need this list be sharable.
>>>> One bolt needs the list for computation.
>>>> Another bolt will periodically update the list.
>>>> In other words, the list is changeable.
>>>>
>>>> I tried passing it in conf but there are some 3rd party libs in the
>>>> wrapper.
>>>>
>>>> Anyway, I also need to improve the topology. Separate bolt to two etc.
>>>> That way, it would be easier to handle that cache.
>>>>
>>>> One more question, I thought about giving hazelcast a try.
>>>> Create local node and then the bolts will have clients.
>>>>
>>>> What do you think?
>>>>
>>>> Sent from my Android phone
>>>> On Dec 4, 2014 9:14 PM, "Kushan Maskey" <
>>>> kushan.maskey@mmillerassociates.com> wrote:
>>>>
>>>>>  Hi Eyal,
>>>>>
>>>>>  I had the similar issue earlier. Static objects are not a good way of
>>>>> passing the variables around in Storm due to clustered nature of the
Storm.
>>>>> You will see null for all the static variables.
>>>>>
>>>>>  This is how I implemented mine, for instance I wanted to pass around
>>>>> the server properties such and host name and username/password. I read
the
>>>>> properties file and then created a map of these properties then set it
into
>>>>> the Config object. Like below.
>>>>>
>>>>>  *In topology:*
>>>>>
>>>>>  Map<Stirng, Stirng> map = new HashMap();
>>>>> map.put("hostname", "localhost");
>>>>>  map.put("username", "testuser");
>>>>>  map.put("password", "testpassord");
>>>>> map.put("port", "1234");
>>>>>
>>>>> Config config = new Config();
>>>>>
>>>>> config.put("MY_MAP", map)
>>>>>  *In Bolts:*
>>>>>
>>>>> public void prepare(Map stormConf, TopologyContext context) {
>>>>>
>>>>> HashMap<Stirng, Stirng> map = stormConfig.get("MY_MAP");
>>>>>
>>>>> }
>>>>>  This how i resolved my problem.Try that if that helps.
>>>>>   --
>>>>> Kushan Maskey
>>>>> 817.403.7500
>>>>> M. Miller & Associates <http://mmillerassociates.com/>
>>>>> kushan.maskey@mmillerassociates.com
>>>>>
>>>>> On Thu, Dec 4, 2014 at 1:01 PM, Eyal Golan <egolan74@gmail.com>
wrote:
>>>>>
>>>>>> Hi all,
>>>>>> I am newbie with Storm and having some issues with static objects
in
>>>>>> the topology.
>>>>>> In the main class, where I set the topology, I also set a static
>>>>>> object. It's actually a wrapper for a collection of strings.
>>>>>> This object needs to he shared between different bolts.
>>>>>> It's big so I want it to be only one instance.
>>>>>>
>>>>>> The problem is that when a bolt tries to get this static instance:
>>>>>> example - MyMain.Shared... It's null. Although I instantiated it
in the
>>>>>> main method.
>>>>>>
>>>>>> I guess I'm doing something wrong.
>>>>>>
>>>>>> Thanks for any assistance.
>>>>>>
>>>>>> Eyal
>>>>>>
>>>>>> Sent from my Android phone
>>>>>>
>>>>>
>>>>>
>

Mime
View raw message