storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stig Rohde Døssing <s...@apache.org>
Subject Re: How to set Zookeeper's data directory & port when running in LocalCluster mode?
Date Sun, 27 Aug 2017 19:31:32 GMT
You can set the Zookeeper configuration through the daemon conf map when
you use a LocalCluster.Builder, which should let you set many of the
options you might normally put in storm.yaml
https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/LocalCluster.java#L173.


I don't believe there is a way to set the data dir for Zookeeper right now.
As you noted, it is set to use the system's temporary directory
https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/testing/InProcessZookeeper.java#L39.
Consider raising an issue if you think we should add a function to set this
(could e.g. be a new function on the LocalCluster.Builder). A PR would
obviously be appreciated if you're up for it :)

I'm not sure what you mean by the LocalCluster(host, port) constructor
being undocumented. What kind of documentation do you feel is missing?

2017-08-27 19:10 GMT+02:00 Alexandre Vermeerbergen <avermeerbergen@gmail.com
>:

> Hello,
>
> I would like to start topologies in LocalCluster mode for testing purposes
> using my own settings for the Zookeeper server started by the LocalCluster
> mode.
>
> For example, I would like Zookeeper to write in
> e:\tmp\exclam\stormlocaldir and use port 123456.
>
> Storm documentation isn't quite verbose about how to set LocalCluster mode
> options, unless I missed the appropriate page (in which please, please let
> me know at which URL I can find how to pass options to LocalCluster mode).
>
> I tried to modify the main() method of  ExclamationTopology classic Storm
> sampe to fix Zookeeer properties, like this:
>
>     public static void main(String[] args) throws Exception {
>         TopologyBuilder builder = new TopologyBuilder();
>
>         builder.setSpout("word", new TestWordSpout(), 10);
>         builder.setBolt("exclaim1", new ExclamationBolt(),
> 3).shuffleGrouping("word");
>         builder.setBolt("exclaim2", new ExclamationBolt(),
> 2).shuffleGrouping("exclaim1");
>
>         Config conf = new Config();
>         conf.setDebug(true);
>
>         conf.put(Config.STORM_LOCAL_DIR, "E:\\tmp\\exclam\\
> stormlocaldir\\");
>         conf.put(Config.STORM_ZOOKEEPER_ROOT, "E:\\tmp\\exclam\\
> stormlocaldir\\");
>         conf.put(Config.STORM_ZOOKEEPER_PORT,12346);
>         LocalCluster cluster = new LocalCluster();
>         cluster.submitTopology("test", conf, builder.createTopology());
>         Utils.sleep(120000);
>         cluster.killTopology("test");
>         cluster.shutdown();
>     }
> }
>
> But to my despair, all data is written into subdirectories
> C:\Users\ave\AppData\Local\Temp\<some UUID> instead of to
> E:\tmp\exclam\stormlocaldir\ subdirectories.
>
> Also, during the program's execution, Zookeeper listens on port 2000
> instead of listening on port 123456 (I checked this: it's not just a wrong
> trace).
>
> I was able to partially by-pass my problem with data directory by setting
> Java's system property which gives the temporary directory path, but this
> is a very ugly by-pass:
>
>     public static void main(String[] args) throws Exception {
>         TopologyBuilder builder = new TopologyBuilder();
>
>         builder.setSpout("word", new TestWordSpout(), 10);
>         builder.setBolt("exclaim1", new ExclamationBolt(),
> 3).shuffleGrouping("word");
>         builder.setBolt("exclaim2", new ExclamationBolt(),
> 2).shuffleGrouping("exclaim1");
>
>         Config conf = new Config();
>         conf.setDebug(true);
>
>         System.setProperty("java.io.tmpdir", "E:\\tmp\\exclam\\
> stormlocaldir\\");
>         conf.put(Config.STORM_ZOOKEEPER_PORT,12346);
>         LocalCluster cluster = new LocalCluster();
>         cluster.submitTopology("test", conf, builder.createTopology());
>         Utils.sleep(120000);
>         cluster.killTopology("test");
>         cluster.shutdown();
>     }
>
> But I found no by-pass to set Zookeeper's port.
>
> I am aware that another by-pass could be to use my own Zookeeper server
> and then use the un-documented LocalCluster(host,port) constructor to start
> a LocalCluster connect to this Zookeeper server, but frankly... isn't there
> a simpler way to setup properties for the Zookeeper server started by
> default constructor of LocalCluster ?
>
> Best regards,
> Alexandre Vermeerbergen
>
>
>

Mime
View raw message