storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Shepherd <...@nexjsystems.com>
Subject RE: Storm cluster only sees 1 worker node at a time - but swtiches between nodes
Date Fri, 06 Feb 2015 16:19:05 GMT
Yes that worked – thank you very much!

From: Harsha [mailto:storm@harsha.io]
Sent: Friday, February 06, 2015 11:12 AM
To: user@storm.apache.org
Subject: Re: Storm cluster only sees 1 worker node at a time - but swtiches between nodes

hmm .. probably. Can you try deleting the storm-local dir on supervisor machines and restart the supervisor daemons.


On Fri, Feb 6, 2015, at 08:07 AM, David Shepherd wrote:

Yes they are.



I just noticed that the supervisor id is the same for both worker-node-1 and worker-node-2.   Could that be because they were created from the same machine image?

David



From: Harsha [mailto:storm@harsha.io]
Sent: Friday, February 06, 2015 11:05 AM
To: user@storm.apache.org<mailto:user@storm.apache.org>
Subject: Re: Storm cluster only sees 1 worker node at a time - but swtiches between nodes



From you earlier nimbus config

storm.zookeeper.servers:

 - "zookeeper-host-1"

    - "zookeeper-host-2"

    - "zookeeper-host-3"



and supervisor storm.yaml config looks like



storm.zookeeper.servers:

 - "zookeeper-1"

     - "zookeeper-2"

     - "zookeeper-3"



Are they both pointing to same zk cluster?

-Harsha





On Fri, Feb 6, 2015, at 07:51 AM, David Shepherd wrote:

Sorry – that yaml is from the nimbus node that has no supervisor jobs for the supervisord process configured.  I have different yaml om the worker nodes.  The only difference in the yaml is the supervisor slot configuration.  Also the worker machine only has the storm supervisor process configured to start in the supervisord configuration.



Nimbus node configuration

---------------------------------

storm.zookeeper.servers:

     - "zookeeper-1"

     - "zookeeper-2"

     - "zookeeper-3"

storm.zookeeper.port: 2181



nimbus.host: "nimbus"

nimbus.thrift.port: 6627

storm.local.dir: "/var/opt/stormtmp"

java.library.path: "/usr/local/lib"



supervisor.slots.ports:



worker.childopts: "-Xmx768m"

nimbus.childopts: "-Xmx512m"

supervisor.childopts: "-Xmx256m"

ui.childopts: "-Xmx512m



Worker node configuration

---------------------------------

storm.zookeeper.servers:

     - "zookeeper-1"

     - "zookeeper-2"

     - "zookeeper-3"

storm.zookeeper.port: 2181



nimbus.host: "nimbus"

nimbus.thrift.port: 6627

storm.local.dir: "/var/opt/stormtmp"

java.library.path: "/usr/local/lib"



supervisor.slots.ports:

     - 6700

     - 6701



worker.childopts: "-Xmx768m"

nimbus.childopts: "-Xmx512m"

supervisor.childopts: "-Xmx256m"

ui.childopts: "-Xmx512m"



On the nimbus node, the nimbus and ui process are running:

storm-nimbus                     RUNNING    pid 2100, uptime 0:00:26

storm-ui                         RUNNING    pid 2094, uptime 0:00:26



On the worker node 1 only the supervisor process is running

storm-supervisor                 RUNNING    pid 2210, uptime 0:00:14



On the worker node 2 only the supervisor process is running

storm-supervisor                 RUNNING    pid 1845, uptime 0:00:12



after starting the process on  worker node 1 the supervisor log is:

2015-02-06T15:20:53.619+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=storm-worker-1.novalocal

2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75

2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation

2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm

2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=storm-worker-1.novalocal

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm

2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/

2015-02-06T15:20:54.305+0000 b.s.d.supervisor [INFO] Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}

2015-02-06T15:20:54.330+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:20:54.416+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:20:54.418+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@457b9c87<mailto:watcher=org.apache.storm.curator.ConnectionState@457b9c87>

2015-02-06T15:20:54.439+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-2/10.0.0.6:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:20:54.445+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-2/10.0.0.6:2181, initiating session

2015-02-06T15:20:54.492+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-2/10.0.0.6:2181, sessionid = 0x14aefe7c7310013, negotiated timeout = 20000

2015-02-06T15:20:54.494+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:20:54.495+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none

2015-02-06T15:20:55.522+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down

2015-02-06T15:20:55.522+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x14aefe7c7310013 closed

2015-02-06T15:20:55.524+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:20:55.525+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:20:55.525+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@5d4e61f7<mailto:watcher=org.apache.storm.curator.ConnectionState@5d4e61f7>

2015-02-06T15:20:55.530+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-1/10.0.0.5:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:20:55.531+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-1/10.0.0.5:2181, initiating session

2015-02-06T15:20:55.540+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-1/10.0.0.5:2181, sessionid = 0x34aefe7a8450011, negotiated timeout = 20000

2015-02-06T15:20:55.540+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:20:55.659+0000 b.s.d.supervisor [INFO] Starting supervisor with id 7d52f803-7363-4397-9df5-fee60479f57f at host storm-worker-1.novalocal



And the storm ui looks like this:

[cid:image001.png@01D041FE.B72813A0]

after starting the process on  worker node 2 the supervisor log is:

2015-02-06T15:39:16.901+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=storm-worker-2.novalocal

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm

2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/

2015-02-06T15:39:16.925+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:39:16.925+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=storm-worker-2.novalocal

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm

2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/

2015-02-06T15:39:17.580+0000 b.s.d.supervisor [INFO] Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}

2015-02-06T15:39:17.604+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:39:17.692+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:39:17.694+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@366186b4<mailto:watcher=org.apache.storm.curator.ConnectionState@366186b4>

2015-02-06T15:39:17.716+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-1/10.0.0.5:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:39:17.721+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-1/10.0.0.5:2181, initiating session

2015-02-06T15:39:17.745+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-1/10.0.0.5:2181, sessionid = 0x34aefe7a8450012, negotiated timeout = 20000

2015-02-06T15:39:17.747+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:39:17.749+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none

2015-02-06T15:39:18.774+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down

2015-02-06T15:39:18.775+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x34aefe7a8450012 closed

2015-02-06T15:39:18.777+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:39:18.777+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:39:18.778+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@4501cefc<mailto:watcher=org.apache.storm.curator.ConnectionState@4501cefc>

2015-02-06T15:39:18.785+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:39:18.786+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181, initiating session

2015-02-06T15:39:18.796+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001d, negotiated timeout = 20000

2015-02-06T15:39:18.796+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:39:18.863+0000 b.s.d.supervisor [INFO] Starting supervisor with id 7d52f803-7363-4397-9df5-fee60479f57f at host storm-worker-2.novalocal



And the storm ui looks like this:



[cid:image002.png@01D041FE.B72813A0]



If I keep refreshing the UI, the host name toggles back and forth between storm-worker-1.novalocal and storm-worker-2.novalocal, the number of supervisors and slots stays and 1 and 2 respectively.



The supervisor.log is the only log file with any content the 2 worker nodes – the access and metric logs are empty.



The supervisor.log on the nimbus node is:

2015-02-06T15:09:48.692+0000 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=nimbus

2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_75

2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation

2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-39-generic

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm

2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/

2015-02-06T15:09:48.740+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=nimbus

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_75

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-39-generic

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm

2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/

2015-02-06T15:09:49.883+0000 b.s.d.nimbus [INFO] Starting Nimbus with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2" "zookeeper-3"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" nil, "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}

2015-02-06T15:09:49.909+0000 b.s.d.nimbus [INFO] Using default scheduler

2015-02-06T15:09:49.931+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:09:50.134+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:09:50.147+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@5d19fbae<mailto:watcher=org.apache.storm.curator.ConnectionState@5d19fbae>

2015-02-06T15:09:50.211+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:09:50.231+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181, initiating session

2015-02-06T15:09:50.246+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001b, negotiated timeout = 20000

2015-02-06T15:09:50.248+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:09:50.250+0000 b.s.zookeeper [INFO] Zookeeper state update: :connected:none

2015-02-06T15:09:51.270+0000 o.a.s.z.ClientCnxn [INFO] EventThread shut down

2015-02-06T15:09:51.271+0000 o.a.s.z.ZooKeeper [INFO] Session: 0x24aefe7a813001b closed

2015-02-06T15:09:51.272+0000 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]

2015-02-06T15:09:51.272+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting

2015-02-06T15:09:51.273+0000 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@7cc1bdd7<mailto:watcher=org.apache.storm.curator.ConnectionState@7cc1bdd7>

2015-02-06T15:09:51.279+0000 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-06T15:09:51.279+0000 o.a.s.z.ClientCnxn [INFO] Socket connection established to zookeeper-3/10.0.0.8:2181, initiating session

2015-02-06T15:09:51.289+0000 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid = 0x24aefe7a813001c, negotiated timeout = 20000

2015-02-06T15:09:51.289+0000 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED

2015-02-06T15:09:51.324+0000 b.s.d.nimbus [INFO] Starting Nimbus server...





From: Harsha [mailto:storm@harsha.io]
Sent: Thursday, February 05, 2015 2:35 PM
To: user@storm.apache.org<mailto:user@storm.apache.org>
Subject: Re: Storm cluster only sees 1 worker node at a time - but swtiches between nodes



David,

         It looks like there might be some confusion between supervisor and workers.



"Zookeeper is running on separate VMs. 1 VM has the nimbus process, and the UI process, and no workers, and the other 2 machines each have a worker with 2 ports"

You have nimbus, UI and two supervisors and each with one worker slot. These workers starts running once you deploy a topology.

Number of workers per supervisor controlled by this config

supervisor.slots.ports:

  - 6700

Since you specified only one port than it means each supervisor can only spawn one worker when a topology deployed. You can add more ports to increase the number of workers per supervisor.



"However when I go to the Storm UI, the cluster summary says there is only 1 supervisor with 2 slots,  and the supervisor summary on has 1 row, but the host name switches between storm-worker-1 and storm-worker-2. It seems to be seeing both machines, but only 1 at a time."



I am assuming you are using same storm.yaml on all the hosts (i.e nimbus, supervisors etc) . In this case based on your config each supervisor should show it has one worker.

You should be able to see two supervisor listed under "Supervisor" section in storm UI.  Do check the logs if your supervisors might be missing connection to zookeeper or crashing! .



Which version of storm you are using. It might help if you can attach screenshots for storm UI.

Thanks,

Harsha







On Thu, Feb 5, 2015, at 11:05 AM, David Shepherd wrote:

I have set up a Storm cluster on 3 vms running on openstack.  Zookeeper is running on separate VMs. 1 VM has the nimbus process, and the UI process, and no workers, and the other 2 machines each have a worker with 2 ports.  The workers were created by provisioning a VM from an image using the openstack tooling, so they are identical.



Everything starts up with no errors in any logs I can fin.  However when I go to the Storm UI, the cluster summary says there is only 1 supervisor with 2 slots,  and the supervisor summary on has 1 row, but the host name switches between storm-worker-1 and storm-worker-2. It seems to be seeing both machines, but only 1 at a time.



The storm.yaml is:



storm.zookeeper.servers:

    - "zookeeper-host-1"

    - "zookeeper-host-2"

    - "zookeeper-host-3"

storm.zookeeper.port: 2181



nimbus.host: "nimbus"

nimbus.thrift.port: 6627

storm.local.dir: "/var/opt/stormtmp"

java.library.path: "/usr/local/lib"



supervisor.slots.ports:

     - 6700

worker.childopts: "-Xmx768m"

nimbus.childopts: "-Xmx512m"

supervisor.childopts: "-Xmx256m"

ui.childopts: "-Xmx512m"



I am totally stumped – can’t find anything on this behavior anywhere online or in any of the books I have.  If anyone else has experienced this and can point me in the right direction it would be much appreciated.



Thanks,



David





Email had 2 attachments:

  *   image001.png
  45k (image/png)
  *   image002.png
  46k (image/png)



Email had 2 attachments:

  *   image001.png
  45k (image/png)
  *   image002.png
  46k (image/png)

Mime
View raw message