storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexandre Vermeerbergen <avermeerber...@gmail.com>
Subject How to set Zookeeper's data directory & port when running in LocalCluster mode?
Date Sun, 27 Aug 2017 17:10:59 GMT
Hello,

I would like to start topologies in LocalCluster mode for testing purposes
using my own settings for the Zookeeper server started by the LocalCluster
mode.

For example, I would like Zookeeper to write in e:\tmp\exclam\stormlocaldir
and use port 123456.

Storm documentation isn't quite verbose about how to set LocalCluster mode
options, unless I missed the appropriate page (in which please, please let
me know at which URL I can find how to pass options to LocalCluster mode).

I tried to modify the main() method of  ExclamationTopology classic Storm
sampe to fix Zookeeer properties, like this:

    public static void main(String[] args) throws Exception {
        TopologyBuilder builder = new TopologyBuilder();

        builder.setSpout("word", new TestWordSpout(), 10);
        builder.setBolt("exclaim1", new ExclamationBolt(),
3).shuffleGrouping("word");
        builder.setBolt("exclaim2", new ExclamationBolt(),
2).shuffleGrouping("exclaim1");

        Config conf = new Config();
        conf.setDebug(true);

        conf.put(Config.STORM_LOCAL_DIR,
"E:\\tmp\\exclam\\stormlocaldir\\");
        conf.put(Config.STORM_ZOOKEEPER_ROOT,
"E:\\tmp\\exclam\\stormlocaldir\\");
        conf.put(Config.STORM_ZOOKEEPER_PORT,12346);
        LocalCluster cluster = new LocalCluster();
        cluster.submitTopology("test", conf, builder.createTopology());
        Utils.sleep(120000);
        cluster.killTopology("test");
        cluster.shutdown();
    }
}

But to my despair, all data is written into subdirectories
C:\Users\ave\AppData\Local\Temp\<some UUID> instead of to
E:\tmp\exclam\stormlocaldir\ subdirectories.

Also, during the program's execution, Zookeeper listens on port 2000
instead of listening on port 123456 (I checked this: it's not just a wrong
trace).

I was able to partially by-pass my problem with data directory by setting
Java's system property which gives the temporary directory path, but this
is a very ugly by-pass:

    public static void main(String[] args) throws Exception {
        TopologyBuilder builder = new TopologyBuilder();

        builder.setSpout("word", new TestWordSpout(), 10);
        builder.setBolt("exclaim1", new ExclamationBolt(),
3).shuffleGrouping("word");
        builder.setBolt("exclaim2", new ExclamationBolt(),
2).shuffleGrouping("exclaim1");

        Config conf = new Config();
        conf.setDebug(true);

        System.setProperty("java.io.tmpdir",
"E:\\tmp\\exclam\\stormlocaldir\\");
        conf.put(Config.STORM_ZOOKEEPER_PORT,12346);
        LocalCluster cluster = new LocalCluster();
        cluster.submitTopology("test", conf, builder.createTopology());
        Utils.sleep(120000);
        cluster.killTopology("test");
        cluster.shutdown();
    }

But I found no by-pass to set Zookeeper's port.

I am aware that another by-pass could be to use my own Zookeeper server and
then use the un-documented LocalCluster(host,port) constructor to start a
LocalCluster connect to this Zookeeper server, but frankly... isn't there a
simpler way to setup properties for the Zookeeper server started by default
constructor of LocalCluster ?

Best regards,
Alexandre Vermeerbergen

Mime
View raw message