Yes that worked – thank you very much!

 

From: Harsha [mailto:storm@harsha.io]
Sent: Friday, February 06, 2015 11:12 AM
To: user@storm.apache.org
Subject: Re: Storm cluster only sees 1 worker node at a time - but swtiches between nodes

 

hmm .. probably. Can you try deleting the storm-local dir on supervisor machines and restart the supervisor daemons.

 

 

On Fri, Feb 6, 2015, at 08:07 AM, David Shepherd wrote:

Yes they are. 

 

I just noticed that the supervisor id is the same for both worker-node-1 and worker-node-2.   Could that be because they were created from the same machine image?


David

 

From: Harsha [mailto:storm@harsha.io]
Sent: Friday, February 06, 2015 11:05 AM
To: user@storm.apache.org
Subject: Re: Storm cluster only sees 1 worker node at a time - but swtiches between nodes

 

From you earlier nimbus config

storm.zookeeper.servers:

 - "zookeeper-host-1"

    - "zookeeper-host-2"

    - "zookeeper-host-3"

 

and supervisor storm.yaml config looks like

 

storm.zookeeper.servers:

 - "zookeeper-1"

     - "zookeeper-2"

     - "zookeeper-3"

 

Are they both pointing to same zk cluster?

-Harsha

 

 

On Fri, Feb 6, 2015, at 07:51 AM, David Shepherd wrote:

Sorry – that yaml is from the nimbus node that has no supervisor jobs for the supervisord process configured.  I have different yaml om the worker nodes.  The only difference in the yaml is the supervisor slot configuration.  Also the worker machine only has the storm supervisor process configured to start in the supervisord configuration.

 

Nimbus node configuration

---------------------------------

storm.zookeeper.servers:

     - "zookeeper-1"

     - "zookeeper-2"

     - "zookeeper-3"

storm.zookeeper.port: 2181

 

nimbus.host: "nimbus"

nimbus.thrift.port: 6627

storm.local.dir: "/var/opt/stormtmp"

java.library.path: "/usr/local/lib"

 

supervisor.slots.ports:

 

worker.childopts: "-Xmx768m"

nimbus.childopts: "-Xmx512m"

supervisor.childopts: "-Xmx256m"

ui.childopts: "-Xmx512m

 

Worker node configuration

---------------------------------

storm.zookeeper.servers:

     - "zookeeper-1"

     - "zookeeper-2"

     - "zookeeper-3"

storm.zookeeper.port: 2181

 

nimbus.host: "nimbus"

nimbus.thrift.port: 6627

storm.local.dir: "/var/opt/stormtmp"

java.library.path: "/usr/local/lib"

 

supervisor.slots.ports:

     - 6700

     - 6701

 

worker.childopts: "-Xmx768m"

nimbus.childopts: "-Xmx512m"

supervisor.childopts: "-Xmx256m"

ui.childopts: "-Xmx512m"

 

On the nimbus node, the nimbus and ui process are running:

storm-nimbus                     RUNNING    pid 2100, uptime 0:00:26

storm-ui                         RUNNING    pid 2094, uptime 0:00:26

 

On the worker node 1 only the supervisor process is running

storm-supervisor                 RUNNING    pid 2210, uptime 0:00:14

 

On the worker node 2 only the supervisor process is running

storm-supervisor                 RUNNING    pid 1845, uptime 0:00:12

 

after starting the process on  worker node 1 the supervisor log is:

2015-02-06T15:20:53.619+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=storm-worker-1.novalocal

2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75

2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation

2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=storm-worker-1.novalocal

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/

2015-02-06T15:20:54.305+0000 b.s.d.supervisor [INFO] Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}

2015-02-06T15:20:54.330+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:20:54.416+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:20:54.418+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@457b9c87

2015-02-06T15:20:54.439+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-2/10.0.0.6:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:20:54.445+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-2/10.0.0.6:2181, initiating session

2015-02-06T15:20:54.492+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-2/10.0.0.6:2181, sessionid = 0x14aefe7c7310013, negotiated timeout = 20000

2015-02-06T15:20:54.494+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:20:54.495+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none

2015-02-06T15:20:55.522+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down

2015-02-06T15:20:55.522+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x14aefe7c7310013 closed

2015-02-06T15:20:55.524+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:20:55.525+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:20:55.525+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@5d4e61f7

2015-02-06T15:20:55.530+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-1/10.0.0.5:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:20:55.531+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-1/10.0.0.5:2181, initiating session

2015-02-06T15:20:55.540+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-1/10.0.0.5:2181, sessionid = 0x34aefe7a8450011, negotiated timeout = 20000

2015-02-06T15:20:55.540+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:20:55.659+0000 b.s.d.supervisor [INFO] Starting supervisor with id 7d52f803-7363-4397-9df5-fee60479f57f at host storm-worker-1.novalocal

 

And the storm ui looks like this:

after starting the process on  worker node 2 the supervisor log is:

2015-02-06T15:39:16.901+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=storm-worker-2.novalocal

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/

2015-02-06T15:39:16.925+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:39:16.925+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=storm-worker-2.novalocal

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/

2015-02-06T15:39:17.580+0000 b.s.d.supervisor [INFO] Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}

2015-02-06T15:39:17.604+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:39:17.692+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:39:17.694+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@366186b4

2015-02-06T15:39:17.716+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-1/10.0.0.5:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:39:17.721+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-1/10.0.0.5:2181, initiating session

2015-02-06T15:39:17.745+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-1/10.0.0.5:2181, sessionid = 0x34aefe7a8450012, negotiated timeout = 20000

2015-02-06T15:39:17.747+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:39:17.749+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none

2015-02-06T15:39:18.774+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down

2015-02-06T15:39:18.775+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x34aefe7a8450012 closed

2015-02-06T15:39:18.777+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:39:18.777+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:39:18.778+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@4501cefc

2015-02-06T15:39:18.785+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:39:18.786+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181, initiating session

2015-02-06T15:39:18.796+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001d, negotiated timeout = 20000

2015-02-06T15:39:18.796+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:39:18.863+0000 b.s.d.supervisor [INFO] Starting supervisor with id 7d52f803-7363-4397-9df5-fee60479f57f at host storm-worker-2.novalocal

 

And the storm ui looks like this:

 

 

If I keep refreshing the UI, the host name toggles back and forth between storm-worker-1.novalocal and storm-worker-2.novalocal, the number of supervisors and slots stays and 1 and 2 respectively.

 

The supervisor.log is the only log file with any content the 2 worker nodes – the access and metric logs are empty.

 

The supervisor.log on the nimbus node is:

2015-02-06T15:09:48.692+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=nimbus

2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75

2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation

2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/

2015-02-06T15:09:48.740+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=nimbus

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/

2015-02-06T15:09:49.883+0000 b.s.d.nimbus [INFO] Starting Nimbus with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" nil, "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}

2015-02-06T15:09:49.909+0000 b.s.d.nimbus [INFO] Using default scheduler

2015-02-06T15:09:49.931+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:09:50.134+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:09:50.147+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@5d19fbae

2015-02-06T15:09:50.211+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:09:50.231+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181, initiating session

2015-02-06T15:09:50.246+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001b, negotiated timeout = 20000

2015-02-06T15:09:50.248+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:09:50.250+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none

2015-02-06T15:09:51.270+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down

2015-02-06T15:09:51.271+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x24aefe7a813001b closed

2015-02-06T15:09:51.272+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:09:51.272+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:09:51.273+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@7cc1bdd7

2015-02-06T15:09:51.279+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:09:51.279+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181, initiating session

2015-02-06T15:09:51.289+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001c, negotiated timeout = 20000

2015-02-06T15:09:51.289+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:09:51.324+0000 b.s.d.nimbus [INFO] Starting Nimbus server...

 

 

From: Harsha [mailto:storm@harsha.io]
Sent: Thursday, February 05, 2015 2:35 PM
To: user@storm.apache.org
Subject: Re: Storm cluster only sees 1 worker node at a time - but swtiches between nodes

 

David,

         It looks like there might be some confusion between supervisor and workers.

 

"Zookeeper is running on separate VMs. 1 VM has the nimbus process, and the UI process, and no workers, and the other 2 machines each have a worker with 2 ports"

You have nimbus, UI and two supervisors and each with one worker slot. These workers starts running once you deploy a topology.

Number of workers per supervisor controlled by this config

supervisor.slots.ports:

  - 6700

Since you specified only one port than it means each supervisor can only spawn one worker when a topology deployed. You can add more ports to increase the number of workers per supervisor.

 

"However when I go to the Storm UI, the cluster summary says there is only 1 supervisor with 2 slots,  and the supervisor summary on has 1 row, but the host name switches between storm-worker-1 and storm-worker-2. It seems to be seeing both machines, but only 1 at a time."

 

I am assuming you are using same storm.yaml on all the hosts (i.e nimbus, supervisors etc) . In this case based on your config each supervisor should show it has one worker.

You should be able to see two supervisor listed under "Supervisor" section in storm UI.  Do check the logs if your supervisors might be missing connection to zookeeper or crashing! .

 

Which version of storm you are using. It might help if you can attach screenshots for storm UI.

Thanks,

Harsha

 

 

 

On Thu, Feb 5, 2015, at 11:05 AM, David Shepherd wrote:

I have set up a Storm cluster on 3 vms running on openstack.  Zookeeper is running on separate VMs. 1 VM has the nimbus process, and the UI process, and no workers, and the other 2 machines each have a worker with 2 ports.  The workers were created by provisioning a VM from an image using the openstack tooling, so they are identical.

 

Everything starts up with no errors in any logs I can fin.  However when I go to the Storm UI, the cluster summary says there is only 1 supervisor with 2 slots,  and the supervisor summary on has 1 row, but the host name switches between storm-worker-1 and storm-worker-2. It seems to be seeing both machines, but only 1 at a time.

 

The storm.yaml is:

 

storm.zookeeper.servers:

    - "zookeeper-host-1"

    - "zookeeper-host-2"

    - "zookeeper-host-3"

storm.zookeeper.port: 2181

 

nimbus.host: "nimbus"

nimbus.thrift.port: 6627

storm.local.dir: "/var/opt/stormtmp"

java.library.path: "/usr/local/lib"

 

supervisor.slots.ports:

     - 6700

worker.childopts: "-Xmx768m"

nimbus.childopts: "-Xmx512m"

supervisor.childopts: "-Xmx256m"

ui.childopts: "-Xmx512m"

 

I am totally stumped – can’t find anything on this behavior anywhere online or in any of the books I have.  If anyone else has experienced this and can point me in the right direction it would be much appreciated.

 

Thanks,

 

David

 

 

Email had 2 attachments:

  • image001.png
      45k (image/png)
  • image002.png
      46k (image/png)

 

Email had 2 attachments: