storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Sharma <deepakmc...@gmail.com>
Subject Re: How do you add a custom class to Config?
Date Thu, 21 Apr 2016 06:47:21 GMT
Hi Navin
If the MongoDatabaseManager is serialized, you should be able to dd it to
the config of the topology.
If not , the class has to be serialized.

Thanks
Deepak

On Thu, Apr 21, 2016 at 11:57 AM, Navin Ipe <navin.ipe@searchlighthealth.com
> wrote:

> Ah ok that's a nice idea, but in my situation it's like this:
> I want to initialize a database connection and also hold some database
> rows and other values in a class. Once these are initialized in a class, I
> don't want any other class to initialize it for the sake of saving memory
> and for referential integrity. It is this initialized object that I want
> all my Spouts to be able to access.
> That's why I was hoping that at least if Config could hold the class, then
> all Spouts and Bolts in the topology would be able to access the
> initialized values, because Config is part of the topology and can be
> accessed in open() and prepare().
>
>
> On Thu, Apr 21, 2016 at 11:08 AM, Abhishek Agarwal <abhishcool@gmail.com>
> wrote:
>
>> In the prepare/open method, you can get the class name and initialize the
>> instance from class name
>>
>> Excuse typos
>> On Apr 21, 2016 9:17 AM, "Navin Ipe" <navin.ipe@searchlighthealth.com>
>> wrote:
>>
>>> Thanks Abhishek, but I don't quite understand what you meant. Do you
>>> mean I pass a class name of a static class and then use it?
>>>
>>> Also took a look at the source code and this is what it says:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> */** * Topology configs are specified as a plain old map. This class
>>> provides a * convenient way to create a topology config map by providing
>>> setter methods for * all the configs that can be set. It also makes it
>>> easier to do things like add * serializations. * * This class also provides
>>> constants for all the configurations possible on * a Storm cluster and
>>> Storm topology. Each constant is paired with an annotation * that defines
>>> the validity criterion of the corresponding field. Default * values for
>>> these configs can be found in defaults.yaml. * * Note that you may put
>>> other configurations in any of the configs. Storm * will ignore anything it
>>> doesn't recognize, but your topologies are free to make * use of them by
>>> reading them in the prepare method of Bolts or the open method of *
>>> Spouts. */public class Config extends HashMap<String, Object> {*
>>>
>>> So if Storm is supposed to ignore anything it does not recognize, then
>>> it's strange it's throwing an error for my serializable class.
>>>
>>> On Thu, Apr 21, 2016 at 12:01 AM, Abhishek Agarwal <abhishcool@gmail.com
>>> > wrote:
>>>
>>>> Pass the class name in config instead of concrete object.
>>>>
>>>> Excuse typos
>>>> On Apr 20, 2016 9:43 PM, "Navin Ipe" <navin.ipe@searchlighthealth.com>
>>>> wrote:
>>>>
>>>>> Does anyone know how to do this? If not adding a custom class, then at
>>>>> least about how to maintain information retrieved from a database in
a
>>>>> single class which all spouts/bolts in all the workers will be able to
>>>>> access?
>>>>>
>>>>> On Tue, Apr 19, 2016 at 2:52 PM, Navin Ipe <
>>>>> navin.ipe@searchlighthealth.com> wrote:
>>>>>
>>>>>> I have this
>>>>>>
>>>>>> Config config = new Config();
>>>>>> MongoDatabaseManager mongoManager = new MongoDatabaseManager();
>>>>>> config.put("MongoManager", mongoManager);
>>>>>>
>>>>>> and MongoDatabaseManager is an empty class:
>>>>>> public class MongoDatabaseManager implements Serializable {}
>>>>>>
>>>>>> But after submitting the topology, I get this error:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails -
>>>>>> Creating a new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043)
with
>>>>>> resources: {"supervisor.cpu.capacity" 400.0,
>>>>>> "supervisor.memory.capacity.mb" 3072.0}14:45:18.321 [main] ERROR
>>>>>> o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Thread Thread[main,5,main]
>>>>>> diedjava.lang.IllegalArgumentException: Topology conf is not
>>>>>> json-serializable    at
>>>>>> org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
>>>>>> ~[storm-core-1.0.0.jar:1.0.0]    at
>>>>>> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
>>>>>> ~[storm-core-1.0.0.jar:1.0.0]    at
>>>>>> org.apache.storm.LocalCluster.submitTopology(Unknown Source)
>>>>>> ~[storm-core-1.0.0.jar:1.0.0]    at
>>>>>> com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
>>>>>>
>>>>>> Why can't I add a custom class to config? The idea here is to store
>>>>>> some info in the class so that all workers can access the information
>>>>>> (because the class is part of the topology) even though they are
part of
>>>>>> different JVM's (the class can be accessed in the open() or prepare()
>>>>>> functions of spouts or bolts, but first, I don't understand why this
crash
>>>>>> is happening).
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Navin
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Navin
>>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Navin
>>>
>>
>
>
> --
> Regards,
> Navin
>



-- 
Thanks
Deepak
www.bigdatabig.com
www.keosha.net

Mime
View raw message