storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Flint, Ben" <bfl...@popcap.com>
Subject Re: Shared static data between bolts
Date Thu, 04 Dec 2014 19:52:21 GMT
I've used Hazelcast for use cases like this with great success.  It makes it relatively straightforward
to implement a distributed data structure.

From: Eyal Golan <egolan74@gmail.com<mailto:egolan74@gmail.com>>
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>>
Date: Thursday, December 4, 2014 at 11:40 AM
To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>>
Subject: Re: Shared static data between bolts


Thanks!
All the answers give me good idea.
Here's why I need this list be sharable.
One bolt needs the list for computation.
Another bolt will periodically update the list.
In other words, the list is changeable.

I tried passing it in conf but there are some 3rd party libs in the wrapper.

Anyway, I also need to improve the topology. Separate bolt to two etc. That way, it would
be easier to handle that cache.

One more question, I thought about giving hazelcast a try.
Create local node and then the bolts will have clients.

What do you think?

Sent from my Android phone

On Dec 4, 2014 9:14 PM, "Kushan Maskey" <kushan.maskey@mmillerassociates.com<mailto:kushan.maskey@mmillerassociates.com>>
wrote:
Hi Eyal,

I had the similar issue earlier. Static objects are not a good way of passing the variables
around in Storm due to clustered nature of the Storm. You will see null for all the static
variables.

This is how I implemented mine, for instance I wanted to pass around the server properties
such and host name and username/password. I read the properties file and then created a map
of these properties then set it into the Config object. Like below.

In topology:

Map<Stirng, Stirng> map = new HashMap();
map.put("hostname", "localhost");
map.put("username", "testuser");
map.put("password", "testpassord");
map.put("port", "1234");

Config config = new Config();

config.put("MY_MAP", map)

In Bolts:

public void prepare(Map stormConf, TopologyContext context) {

HashMap<Stirng, Stirng> map = stormConfig.get("MY_MAP");

}

This how i resolved my problem.Try that if that helps.
--
Kushan Maskey
817.403.7500<tel:817.403.7500>
M. Miller & Associates<http://mmillerassociates.com/>
kushan.maskey@mmillerassociates.com<mailto:kushan.maskey@mmillerassociates.com>

On Thu, Dec 4, 2014 at 1:01 PM, Eyal Golan <egolan74@gmail.com<mailto:egolan74@gmail.com>>
wrote:

Hi all,
I am newbie with Storm and having some issues with static objects in the topology.
In the main class, where I set the topology, I also set a static object. It's actually a wrapper
for a collection of strings.
This object needs to he shared between different bolts.
It's big so I want it to be only one instance.

The problem is that when a bolt tries to get this static instance: example - MyMain.Shared...
It's null. Although I instantiated it in the main method.

I guess I'm doing something wrong.

Thanks for any assistance.

Eyal

Sent from my Android phone


Mime
View raw message