storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navin Ipe <navin....@searchlighthealth.com>
Subject Re: How do you add a custom class to Config?
Date Thu, 21 Apr 2016 08:38:04 GMT
@Deepak: In the first email of this thread I had shown the code I was
trying to use. MongoDatabaseManager is serialized and I've even tried using
it as an empty class but it didn't work, which is why I asked on the
mailing list. Since it's a tiny piece of code, you could also try it out on
your system.

Config config = new Config();
MongoDatabaseManager mongoManager = new MongoDatabaseManager();
config.put("MongoManager", mongoManager);

and MongoDatabaseManager is an empty class:
public class MongoDatabaseManager implements Serializable {}

But after submitting the topology, I get this error:






*14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails - Creating a
new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) with resources:
{"supervisor.cpu.capacity" 400.0, "supervisor.memory.capacity.mb"
3072.0}14:45:18.321 [main] ERROR o.a.s.s.o.a.z.s.NIOServerCnxnFactory -
Thread Thread[main,5,main] diedjava.lang.IllegalArgumentException: Topology
conf is not json-serializable    at
org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
~[storm-core-1.0.0.jar:1.0.0]    at
org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
~[storm-core-1.0.0.jar:1.0.0]    at
org.apache.storm.LocalCluster.submitTopology(Unknown Source)
~[storm-core-1.0.0.jar:1.0.0]    at
com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*

On Thu, Apr 21, 2016 at 12:17 PM, Deepak Sharma <deepakmca05@gmail.com>
wrote:

> Hi Navin
> If the MongoDatabaseManager is serialized, you should be able to dd it to
> the config of the topology.
> If not , the class has to be serialized.
>
> Thanks
> Deepak
>
> On Thu, Apr 21, 2016 at 11:57 AM, Navin Ipe <
> navin.ipe@searchlighthealth.com> wrote:
>
>> Ah ok that's a nice idea, but in my situation it's like this:
>> I want to initialize a database connection and also hold some database
>> rows and other values in a class. Once these are initialized in a class, I
>> don't want any other class to initialize it for the sake of saving memory
>> and for referential integrity. It is this initialized object that I want
>> all my Spouts to be able to access.
>> That's why I was hoping that at least if Config could hold the class,
>> then all Spouts and Bolts in the topology would be able to access the
>> initialized values, because Config is part of the topology and can be
>> accessed in open() and prepare().
>>
>>
>> On Thu, Apr 21, 2016 at 11:08 AM, Abhishek Agarwal <abhishcool@gmail.com>
>> wrote:
>>
>>> In the prepare/open method, you can get the class name and initialize
>>> the instance from class name
>>>
>>> Excuse typos
>>> On Apr 21, 2016 9:17 AM, "Navin Ipe" <navin.ipe@searchlighthealth.com>
>>> wrote:
>>>
>>>> Thanks Abhishek, but I don't quite understand what you meant. Do you
>>>> mean I pass a class name of a static class and then use it?
>>>>
>>>> Also took a look at the source code and this is what it says:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> */** * Topology configs are specified as a plain old map. This class
>>>> provides a * convenient way to create a topology config map by providing
>>>> setter methods for * all the configs that can be set. It also makes it
>>>> easier to do things like add * serializations. * * This class also provides
>>>> constants for all the configurations possible on * a Storm cluster and
>>>> Storm topology. Each constant is paired with an annotation * that defines
>>>> the validity criterion of the corresponding field. Default * values for
>>>> these configs can be found in defaults.yaml. * * Note that you may put
>>>> other configurations in any of the configs. Storm * will ignore anything
it
>>>> doesn't recognize, but your topologies are free to make * use of them by
>>>> reading them in the prepare method of Bolts or the open method of *
>>>> Spouts. */public class Config extends HashMap<String, Object> {*
>>>>
>>>> So if Storm is supposed to ignore anything it does not recognize, then
>>>> it's strange it's throwing an error for my serializable class.
>>>>
>>>> On Thu, Apr 21, 2016 at 12:01 AM, Abhishek Agarwal <
>>>> abhishcool@gmail.com> wrote:
>>>>
>>>>> Pass the class name in config instead of concrete object.
>>>>>
>>>>> Excuse typos
>>>>> On Apr 20, 2016 9:43 PM, "Navin Ipe" <navin.ipe@searchlighthealth.com>
>>>>> wrote:
>>>>>
>>>>>> Does anyone know how to do this? If not adding a custom class, then
>>>>>> at least about how to maintain information retrieved from a database
in a
>>>>>> single class which all spouts/bolts in all the workers will be able
to
>>>>>> access?
>>>>>>
>>>>>> On Tue, Apr 19, 2016 at 2:52 PM, Navin Ipe <
>>>>>> navin.ipe@searchlighthealth.com> wrote:
>>>>>>
>>>>>>> I have this
>>>>>>>
>>>>>>> Config config = new Config();
>>>>>>> MongoDatabaseManager mongoManager = new MongoDatabaseManager();
>>>>>>> config.put("MongoManager", mongoManager);
>>>>>>>
>>>>>>> and MongoDatabaseManager is an empty class:
>>>>>>> public class MongoDatabaseManager implements Serializable {}
>>>>>>>
>>>>>>> But after submitting the topology, I get this error:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails
-
>>>>>>> Creating a new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043)
with
>>>>>>> resources: {"supervisor.cpu.capacity" 400.0,
>>>>>>> "supervisor.memory.capacity.mb" 3072.0}14:45:18.321 [main] ERROR
>>>>>>> o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Thread Thread[main,5,main]
>>>>>>> diedjava.lang.IllegalArgumentException: Topology conf is not
>>>>>>> json-serializable    at
>>>>>>> org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
>>>>>>> ~[storm-core-1.0.0.jar:1.0.0]    at
>>>>>>> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
>>>>>>> ~[storm-core-1.0.0.jar:1.0.0]    at
>>>>>>> org.apache.storm.LocalCluster.submitTopology(Unknown Source)
>>>>>>> ~[storm-core-1.0.0.jar:1.0.0]    at
>>>>>>> com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
>>>>>>>
>>>>>>> Why can't I add a custom class to config? The idea here is to
store
>>>>>>> some info in the class so that all workers can access the information
>>>>>>> (because the class is part of the topology) even though they
are part of
>>>>>>> different JVM's (the class can be accessed in the open() or prepare()
>>>>>>> functions of spouts or bolts, but first, I don't understand why
this crash
>>>>>>> is happening).
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Navin
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Navin
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Navin
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Navin
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>



-- 
Regards,
Navin

Mime
View raw message