storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsha <st...@harsha.io>
Subject Re: Storm cluster only sees 1 worker node at a time - but swtiches between nodes
Date Fri, 06 Feb 2015 16:05:09 GMT

>From you earlier nimbus config storm.zookeeper.servers: -
"zookeeper-host-1"

- "zookeeper-host-2"

- "zookeeper-host-3"


and supervisor storm.yaml config looks like

storm.zookeeper.servers: - "zookeeper-1" - "zookeeper-2"

- "zookeeper-3"

Are they both pointing to same zk cluster? -Harsha


On Fri, Feb 6, 2015, at 07:51 AM, David Shepherd wrote:
> Sorry – that yaml is from the nimbus node that has no supervisor jobs
> for the supervisord process configured. I have different yaml om the
> worker nodes. The
 only difference in the yaml is the supervisor slot configuration. Also
 the worker machine only has the storm supervisor process configured to
 start in the supervisord configuration.


>


> Nimbus node configuration


> ---------------------------------


> storm.zookeeper.servers:


> - "zookeeper-1"


> - "zookeeper-2"


> - "zookeeper-3"


> storm.zookeeper.port: 2181


>


> nimbus.host: "nimbus"


> nimbus.thrift.port: 6627


> storm.local.dir: "/var/opt/stormtmp"


> java.library.path: "/usr/local/lib"


>


> supervisor.slots.ports:


>


> worker.childopts: "-Xmx768m"


> nimbus.childopts: "-Xmx512m"


> supervisor.childopts: "-Xmx256m"


> ui.childopts: "-Xmx512m


>


> Worker node configuration


> ---------------------------------


> storm.zookeeper.servers:


> - "zookeeper-1"


> - "zookeeper-2"


> - "zookeeper-3"


> storm.zookeeper.port: 2181


>


> nimbus.host: "nimbus"


> nimbus.thrift.port: 6627


> storm.local.dir: "/var/opt/stormtmp"


> java.library.path: "/usr/local/lib"


>


> supervisor.slots.ports:


> - 6700


> - 6701


>


> worker.childopts: "-Xmx768m"


> nimbus.childopts: "-Xmx512m"


> supervisor.childopts: "-Xmx256m"


> ui.childopts: "-Xmx512m"


>


> On the nimbus node, the nimbus and ui process are running:


> storm-nimbus RUNNING pid 2100, uptime 0:00:26


> storm-ui RUNNING pid 2094, uptime 0:00:26


>


> On the worker node 1 only the supervisor process is running


> storm-supervisor RUNNING pid 2210, uptime 0:00:14


>


> On the worker node 2 only the supervisor process is running


> storm-supervisor RUNNING pid 1845, uptime 0:00:12


>


> after starting the process on worker node 1 the supervisor log is:


> 2015-02-06T15:20:53.619+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> 09:09 GMT


> 2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:host.name=storm-worker-1.novalocal


> 2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.version=1.7.0_75


> 2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.vendor=Oracle Corporation


> 2015-02-06T15:20:53.625+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre


> 2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf


> 2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.library.path=/usr/local/lib


> 2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.io.tmpdir=/tmp


> 2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.compiler=<NA>


> 2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:os.name=Linux


> 2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:os.arch=amd64


> 2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:os.version=3.13.0-39-generic


> 2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:user.name=storm


> 2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:user.home=/home/storm


> 2015-02-06T15:20:53.626+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:user.dir=/


> 2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> 09:09 GMT


> 2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:host.name=storm-worker-1.novalocal


> 2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.version=1.7.0_75


> 2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.vendor=Oracle Corporation


> 2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre


> 2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf


> 2015-02-06T15:20:53.643+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.library.path=/usr/local/lib


> 2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.io.tmpdir=/tmp


> 2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.compiler=<NA>


> 2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:os.name=Linux


> 2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:os.arch=amd64


> 2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:os.version=3.13.0-39-generic


> 2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:user.name=storm


> 2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:user.home=/home/storm


> 2015-02-06T15:20:53.644+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:user.dir=/


> 2015-02-06T15:20:54.305+0000 b.s.d.supervisor [INFO] Starting
> Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper",
> "topology.tick.tuple.freq.secs"
 nil, "topology.builtin.metrics.bucket.size.secs" 60,
 "topology.fall.back.on.java.serialization" true,
 "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
 "topology.skip.missing.kryo.registrations" false,
 "storm.messaging.netty.client_worker_threads" 1, "ui.childopts"
 "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign"
 true, "topology.trident.batch.emit.interval.millis" 500,
 "storm.messaging.netty.flush.check.interval.ms" 10,
 "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m",
 "java.library.path" "/usr/local/lib",
 "topology.executor.send.buffer.size" 1024, "storm.local.dir"
 "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880,
 "supervisor.worker.start.timeout.secs" 120,
 "topology.enable.message.timeouts" true,
 "nimbus.cleanup.inbox.freq.secs" 600,
 "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
 "storm.meta.serialization.delegate"
 "backtype.storm.serialization.DefaultSerializationDelegate",
 "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus",
 "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
 "transactional.zookeeper.port" nil,
 "topology.executor.receive.buffer.size" 1024,
 "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm",
 "storm.zookeeper.retry.intervalceiling.millis" 30000,
 "supervisor.enable" true, "storm.messaging.netty.server_worker_threads"
 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2"
 "zookeeper-3"], "transactional.zookeeper.root" "/transactional",
 "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024,
 "topology.worker.childopts" nil, "drpc.queue.size" 128,
 "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5,
 "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port"
 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts"
 "-Xmx768m", "topology.receiver.buffer.size" 8,
 "task.heartbeat.frequency.secs" 3, "topology.tasks" nil,
 "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy"
 "backtype.storm.spout.SleepSpoutWaitStrategy",
 "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending"
 nil, "storm.zookeeper.retry.interval" 1000,
 "topology.sleep.spout.wait.strategy.time.ms" 1,
 "nimbus.topology.validator"
 "backtype.storm.nimbus.DefaultTopologyValidator",
 "supervisor.slots.ports" [6700 6701], "topology.environment" nil,
 "topology.debug" false, "nimbus.task.launch.secs" 120,
 "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs"
 30, "task.refresh.poll.secs" 10, "topology.workers" 1,
 "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627,
 "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1,
 "topology.tuple.serializer"
 "backtype.storm.serialization.types.ListDelegateSerializer",
 "topology.disruptor.wait.strategy"
 "com.lmax.disruptor.BlockingWaitStrategy",
 "topology.multilang.serializer"
 "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs"
 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
 "backtype.storm.serialization.DefaultKryoFactory",
 "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1,
 "storm.zookeeper.retry.times" 5,
 "topology.worker.receiver.thread.count" 1, "storm.thrift.transport"
 "backtype.storm.security.auth.SimpleTransportPlugin",
 "topology.state.synchronization.timeout.secs" 60,
 "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
 600, "storm.messaging.transport"
 "backtype.storm.messaging.netty.Context", "logviewer.appender.name"
 "A1", "storm.messaging.netty.max_wait_ms" 1000,
 "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false,
 "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode"
 "distributed", "topology.max.task.parallelism" nil,
 "storm.messaging.netty.transfer.batch.size" 262144,
 "topology.classpath" nil}


> 2015-02-06T15:20:54.330+0000 b.s.u.StormBoundedExponentialBackoffRetry
> [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the
> maxRetries [5]


> 2015-02-06T15:20:54.416+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO]
> Starting


> 2015-02-06T15:20:54.418+0000 o.a.s.z.ZooKeeper [INFO] Initiating
> client connection,
> connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
> sessionTimeout=20000
 watcher=org.apache.storm.curator.ConnectionState@457b9c87


> 2015-02-06T15:20:54.439+0000 o.a.s.z.ClientCnxn [INFO] Opening socket
> connection to server zookeeper-2/10.0.0.6:2181. Will not attempt to
> authenticate using
 SASL (unknown error)


> 2015-02-06T15:20:54.445+0000 o.a.s.z.ClientCnxn [INFO] Socket
> connection established to zookeeper-2/10.0.0.6:2181, initiating
> session


> 2015-02-06T15:20:54.492+0000 o.a.s.z.ClientCnxn [INFO] Session
> establishment complete on server zookeeper-2/10.0.0.6:2181, sessionid
> = 0x14aefe7c7310013, negotiated
 timeout = 20000


> 2015-02-06T15:20:54.494+0000 o.a.s.c.f.s.ConnectionStateManager [INFO]
> State change: CONNECTED


> 2015-02-06T15:20:54.495+0000 b.s.zookeeper [INFO] Zookeeper state
> update: :connected:none


> 2015-02-06T15:20:55.522+0000 o.a.s.z.ClientCnxn [INFO] EventThread
> shut down


> 2015-02-06T15:20:55.522+0000 o.a.s.z.ZooKeeper [INFO] Session:
> 0x14aefe7c7310013 closed


> 2015-02-06T15:20:55.524+0000 b.s.u.StormBoundedExponentialBackoffRetry
> [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the
> maxRetries [5]


> 2015-02-06T15:20:55.525+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO]
> Starting


> 2015-02-06T15:20:55.525+0000 o.a.s.z.ZooKeeper [INFO] Initiating
> client connection,
> connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm
> sessionTimeout=20000
 watcher=org.apache.storm.curator.ConnectionState@5d4e61f7


> 2015-02-06T15:20:55.530+0000 o.a.s.z.ClientCnxn [INFO] Opening socket
> connection to server zookeeper-1/10.0.0.5:2181. Will not attempt to
> authenticate using
 SASL (unknown error)


> 2015-02-06T15:20:55.531+0000 o.a.s.z.ClientCnxn [INFO] Socket
> connection established to zookeeper-1/10.0.0.5:2181, initiating
> session


> 2015-02-06T15:20:55.540+0000 o.a.s.z.ClientCnxn [INFO] Session
> establishment complete on server zookeeper-1/10.0.0.5:2181, sessionid
> = 0x34aefe7a8450011, negotiated
 timeout = 20000


> 2015-02-06T15:20:55.540+0000 o.a.s.c.f.s.ConnectionStateManager [INFO]
> State change: CONNECTED


> 2015-02-06T15:20:55.659+0000 b.s.d.supervisor [INFO] Starting
> supervisor with id 7d52f803-7363-4397-9df5-fee60479f57f at host
> storm-worker-1.novalocal


>


> And the storm ui looks like this:


>


> after starting the process on worker node 2 the supervisor log is:


> 2015-02-06T15:39:16.901+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> 09:09 GMT


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:host.name=storm-worker-2.novalocal


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.version=1.7.0_75


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.vendor=Oracle Corporation


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.library.path=/usr/local/lib


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.io.tmpdir=/tmp


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.compiler=<NA>


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:os.name=Linux


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:os.arch=amd64


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:os.version=3.13.0-39-generic


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:user.name=storm


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:user.home=/home/storm


> 2015-02-06T15:39:16.908+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:user.dir=/


> 2015-02-06T15:39:16.925+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> 09:09 GMT


> 2015-02-06T15:39:16.925+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:host.name=storm-worker-2.novalocal


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.version=1.7.0_75


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.vendor=Oracle Corporation


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.library.path=/usr/local/lib


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.io.tmpdir=/tmp


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.compiler=<NA>


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:os.name=Linux


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:os.arch=amd64


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:os.version=3.13.0-39-generic


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:user.name=storm


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:user.home=/home/storm


> 2015-02-06T15:39:16.926+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:user.dir=/


> 2015-02-06T15:39:17.580+0000 b.s.d.supervisor [INFO] Starting
> Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper",
> "topology.tick.tuple.freq.secs"
 nil, "topology.builtin.metrics.bucket.size.secs" 60,
 "topology.fall.back.on.java.serialization" true,
 "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
 "topology.skip.missing.kryo.registrations" false,
 "storm.messaging.netty.client_worker_threads" 1, "ui.childopts"
 "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign"
 true, "topology.trident.batch.emit.interval.millis" 500,
 "storm.messaging.netty.flush.check.interval.ms" 10,
 "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m",
 "java.library.path" "/usr/local/lib",
 "topology.executor.send.buffer.size" 1024, "storm.local.dir"
 "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880,
 "supervisor.worker.start.timeout.secs" 120,
 "topology.enable.message.timeouts" true,
 "nimbus.cleanup.inbox.freq.secs" 600,
 "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
 "storm.meta.serialization.delegate"
 "backtype.storm.serialization.DefaultSerializationDelegate",
 "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus",
 "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
 "transactional.zookeeper.port" nil,
 "topology.executor.receive.buffer.size" 1024,
 "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm",
 "storm.zookeeper.retry.intervalceiling.millis" 30000,
 "supervisor.enable" true, "storm.messaging.netty.server_worker_threads"
 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2"
 "zookeeper-3"], "transactional.zookeeper.root" "/transactional",
 "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024,
 "topology.worker.childopts" nil, "drpc.queue.size" 128,
 "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5,
 "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port"
 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts"
 "-Xmx768m", "topology.receiver.buffer.size" 8,
 "task.heartbeat.frequency.secs" 3, "topology.tasks" nil,
 "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy"
 "backtype.storm.spout.SleepSpoutWaitStrategy",
 "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending"
 nil, "storm.zookeeper.retry.interval" 1000,
 "topology.sleep.spout.wait.strategy.time.ms" 1,
 "nimbus.topology.validator"
 "backtype.storm.nimbus.DefaultTopologyValidator",
 "supervisor.slots.ports" [6700 6701], "topology.environment" nil,
 "topology.debug" false, "nimbus.task.launch.secs" 120,
 "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs"
 30, "task.refresh.poll.secs" 10, "topology.workers" 1,
 "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627,
 "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1,
 "topology.tuple.serializer"
 "backtype.storm.serialization.types.ListDelegateSerializer",
 "topology.disruptor.wait.strategy"
 "com.lmax.disruptor.BlockingWaitStrategy",
 "topology.multilang.serializer"
 "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs"
 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
 "backtype.storm.serialization.DefaultKryoFactory",
 "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1,
 "storm.zookeeper.retry.times" 5,
 "topology.worker.receiver.thread.count" 1, "storm.thrift.transport"
 "backtype.storm.security.auth.SimpleTransportPlugin",
 "topology.state.synchronization.timeout.secs" 60,
 "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
 600, "storm.messaging.transport"
 "backtype.storm.messaging.netty.Context", "logviewer.appender.name"
 "A1", "storm.messaging.netty.max_wait_ms" 1000,
 "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false,
 "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode"
 "distributed", "topology.max.task.parallelism" nil,
 "storm.messaging.netty.transfer.batch.size" 262144,
 "topology.classpath" nil}


> 2015-02-06T15:39:17.604+0000 b.s.u.StormBoundedExponentialBackoffRetry
> [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the
> maxRetries [5]


> 2015-02-06T15:39:17.692+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO]
> Starting


> 2015-02-06T15:39:17.694+0000 o.a.s.z.ZooKeeper [INFO] Initiating
> client connection,
> connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
> sessionTimeout=20000
 watcher=org.apache.storm.curator.ConnectionState@366186b4


> 2015-02-06T15:39:17.716+0000 o.a.s.z.ClientCnxn [INFO] Opening socket
> connection to server zookeeper-1/10.0.0.5:2181. Will not attempt to
> authenticate using
 SASL (unknown error)


> 2015-02-06T15:39:17.721+0000 o.a.s.z.ClientCnxn [INFO] Socket
> connection established to zookeeper-1/10.0.0.5:2181, initiating
> session


> 2015-02-06T15:39:17.745+0000 o.a.s.z.ClientCnxn [INFO] Session
> establishment complete on server zookeeper-1/10.0.0.5:2181, sessionid
> = 0x34aefe7a8450012, negotiated
 timeout = 20000


> 2015-02-06T15:39:17.747+0000 o.a.s.c.f.s.ConnectionStateManager [INFO]
> State change: CONNECTED


> 2015-02-06T15:39:17.749+0000 b.s.zookeeper [INFO] Zookeeper state
> update: :connected:none


> 2015-02-06T15:39:18.774+0000 o.a.s.z.ClientCnxn [INFO] EventThread
> shut down


> 2015-02-06T15:39:18.775+0000 o.a.s.z.ZooKeeper [INFO] Session:
> 0x34aefe7a8450012 closed


> 2015-02-06T15:39:18.777+0000 b.s.u.StormBoundedExponentialBackoffRetry
> [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the
> maxRetries [5]


> 2015-02-06T15:39:18.777+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO]
> Starting


> 2015-02-06T15:39:18.778+0000 o.a.s.z.ZooKeeper [INFO] Initiating
> client connection,
> connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm
> sessionTimeout=20000
 watcher=org.apache.storm.curator.ConnectionState@4501cefc


> 2015-02-06T15:39:18.785+0000 o.a.s.z.ClientCnxn [INFO] Opening socket
> connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to
> authenticate using
 SASL (unknown error)


> 2015-02-06T15:39:18.786+0000 o.a.s.z.ClientCnxn [INFO] Socket
> connection established to zookeeper-3/10.0.0.8:2181, initiating
> session


> 2015-02-06T15:39:18.796+0000 o.a.s.z.ClientCnxn [INFO] Session
> establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid
> = 0x24aefe7a813001d, negotiated
 timeout = 20000


> 2015-02-06T15:39:18.796+0000 o.a.s.c.f.s.ConnectionStateManager [INFO]
> State change: CONNECTED


> 2015-02-06T15:39:18.863+0000 b.s.d.supervisor [INFO] Starting
> supervisor with id 7d52f803-7363-4397-9df5-fee60479f57f at host
> storm-worker-2.novalocal


>


> And the storm ui looks like this:


>


>


>


> If I keep refreshing the UI, the host name toggles back and forth
> between storm-worker-1.novalocal and storm-worker-2.novalocal, the
> number of supervisors and
 slots stays and 1 and 2 respectively.


>


> The supervisor.log is the only log file with any content the 2 worker
> nodes – the access and metric logs are empty.


>


> The supervisor.log on the nimbus node is:


> 2015-02-06T15:09:48.692+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> 09:09 GMT


> 2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:host.name=nimbus


> 2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.version=1.7.0_75


> 2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.vendor=Oracle Corporation


> 2015-02-06T15:09:48.698+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre


> 2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf


> 2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.library.path=/usr/local/lib


> 2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.io.tmpdir=/tmp


> 2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:java.compiler=<NA>


> 2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:os.name=Linux


> 2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:os.arch=amd64


> 2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:os.version=3.13.0-39-generic


> 2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:user.name=storm


> 2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:user.home=/home/storm


> 2015-02-06T15:09:48.699+0000 o.a.s.z.ZooKeeper [INFO] Client
> environment:user.dir=/


> 2015-02-06T15:09:48.740+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> 09:09 GMT


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:host.name=nimbus


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.version=1.7.0_75


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.vendor=Oracle Corporation


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.class.path=/usr/local/lib/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/lib/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/lib/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/lib/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/lib/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/lib/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/lib/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/lib/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/lib/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/lib/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/lib/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/lib/apache-storm-0.9.3/conf


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.library.path=/usr/local/lib


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.io.tmpdir=/tmp


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:java.compiler=<NA>


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:os.name=Linux


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:os.arch=amd64


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:os.version=3.13.0-39-generic


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:user.name=storm


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:user.home=/home/storm


> 2015-02-06T15:09:48.741+0000 o.a.s.z.s.ZooKeeperServer [INFO] Server
> environment:user.dir=/


> 2015-02-06T15:09:49.883+0000 b.s.d.nimbus [INFO] Starting Nimbus with
> conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper",
> "topology.tick.tuple.freq.secs"
 nil, "topology.builtin.metrics.bucket.size.secs" 60,
 "topology.fall.back.on.java.serialization" true,
 "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
 "topology.skip.missing.kryo.registrations" false,
 "storm.messaging.netty.client_worker_threads" 1, "ui.childopts"
 "-Xmx512m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign"
 true, "topology.trident.batch.emit.interval.millis" 500,
 "storm.messaging.netty.flush.check.interval.ms" 10,
 "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m",
 "java.library.path" "/usr/local/lib",
 "topology.executor.send.buffer.size" 1024, "storm.local.dir"
 "/var/opt/stormtmp", "storm.messaging.netty.buffer_size" 5242880,
 "supervisor.worker.start.timeout.secs" 120,
 "topology.enable.message.timeouts" true,
 "nimbus.cleanup.inbox.freq.secs" 600,
 "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
 "storm.meta.serialization.delegate"
 "backtype.storm.serialization.DefaultSerializationDelegate",
 "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "nimbus",
 "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
 "transactional.zookeeper.port" nil,
 "topology.executor.receive.buffer.size" 1024,
 "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm",
 "storm.zookeeper.retry.intervalceiling.millis" 30000,
 "supervisor.enable" true, "storm.messaging.netty.server_worker_threads"
 1, "storm.zookeeper.servers" ["zookeeper-1" "zookeeper-2"
 "zookeeper-3"], "transactional.zookeeper.root" "/transactional",
 "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024,
 "topology.worker.childopts" nil, "drpc.queue.size" 128,
 "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5,
 "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port"
 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts"
 "-Xmx768m", "topology.receiver.buffer.size" 8,
 "task.heartbeat.frequency.secs" 3, "topology.tasks" nil,
 "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy"
 "backtype.storm.spout.SleepSpoutWaitStrategy",
 "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending"
 nil, "storm.zookeeper.retry.interval" 1000,
 "topology.sleep.spout.wait.strategy.time.ms" 1,
 "nimbus.topology.validator"
 "backtype.storm.nimbus.DefaultTopologyValidator",
 "supervisor.slots.ports" nil, "topology.environment" nil,
 "topology.debug" false, "nimbus.task.launch.secs" 120,
 "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs"
 30, "task.refresh.poll.secs" 10, "topology.workers" 1,
 "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627,
 "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1,
 "topology.tuple.serializer"
 "backtype.storm.serialization.types.ListDelegateSerializer",
 "topology.disruptor.wait.strategy"
 "com.lmax.disruptor.BlockingWaitStrategy",
 "topology.multilang.serializer"
 "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs"
 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
 "backtype.storm.serialization.DefaultKryoFactory",
 "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1,
 "storm.zookeeper.retry.times" 5,
 "topology.worker.receiver.thread.count" 1, "storm.thrift.transport"
 "backtype.storm.security.auth.SimpleTransportPlugin",
 "topology.state.synchronization.timeout.secs" 60,
 "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
 600, "storm.messaging.transport"
 "backtype.storm.messaging.netty.Context", "logviewer.appender.name"
 "A1", "storm.messaging.netty.max_wait_ms" 1000,
 "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false,
 "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode"
 "distributed", "topology.max.task.parallelism" nil,
 "storm.messaging.netty.transfer.batch.size" 262144,
 "topology.classpath" nil}


> 2015-02-06T15:09:49.909+0000 b.s.d.nimbus [INFO] Using default
> scheduler


> 2015-02-06T15:09:49.931+0000 b.s.u.StormBoundedExponentialBackoffRetry
> [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the
> maxRetries [5]


> 2015-02-06T15:09:50.134+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO]
> Starting


> 2015-02-06T15:09:50.147+0000 o.a.s.z.ZooKeeper [INFO] Initiating
> client connection,
> connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
> sessionTimeout=20000
 watcher=org.apache.storm.curator.ConnectionState@5d19fbae


> 2015-02-06T15:09:50.211+0000 o.a.s.z.ClientCnxn [INFO] Opening socket
> connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to
> authenticate using
 SASL (unknown error)


> 2015-02-06T15:09:50.231+0000 o.a.s.z.ClientCnxn [INFO] Socket
> connection established to zookeeper-3/10.0.0.8:2181, initiating
> session


> 2015-02-06T15:09:50.246+0000 o.a.s.z.ClientCnxn [INFO] Session
> establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid
> = 0x24aefe7a813001b, negotiated
 timeout = 20000


> 2015-02-06T15:09:50.248+0000 o.a.s.c.f.s.ConnectionStateManager [INFO]
> State change: CONNECTED


> 2015-02-06T15:09:50.250+0000 b.s.zookeeper [INFO] Zookeeper state
> update: :connected:none


> 2015-02-06T15:09:51.270+0000 o.a.s.z.ClientCnxn [INFO] EventThread
> shut down


> 2015-02-06T15:09:51.271+0000 o.a.s.z.ZooKeeper [INFO] Session:
> 0x24aefe7a813001b closed


> 2015-02-06T15:09:51.272+0000 b.s.u.StormBoundedExponentialBackoffRetry
> [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the
> maxRetries [5]


> 2015-02-06T15:09:51.272+0000 o.a.s.c.f.i.CuratorFrameworkImpl [INFO]
> Starting


> 2015-02-06T15:09:51.273+0000 o.a.s.z.ZooKeeper [INFO] Initiating
> client connection,
> connectString=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/storm
> sessionTimeout=20000
 watcher=org.apache.storm.curator.ConnectionState@7cc1bdd7


> 2015-02-06T15:09:51.279+0000 o.a.s.z.ClientCnxn [INFO] Opening socket
> connection to server zookeeper-3/10.0.0.8:2181. Will not attempt to
> authenticate using
 SASL (unknown error)


> 2015-02-06T15:09:51.279+0000 o.a.s.z.ClientCnxn [INFO] Socket
> connection established to zookeeper-3/10.0.0.8:2181, initiating
> session


> 2015-02-06T15:09:51.289+0000 o.a.s.z.ClientCnxn [INFO] Session
> establishment complete on server zookeeper-3/10.0.0.8:2181, sessionid
> = 0x24aefe7a813001c, negotiated
 timeout = 20000


> 2015-02-06T15:09:51.289+0000 o.a.s.c.f.s.ConnectionStateManager [INFO]
> State change: CONNECTED


> 2015-02-06T15:09:51.324+0000 b.s.d.nimbus [INFO] Starting Nimbus
> server...


>


>


> *From:* Harsha [mailto:storm@harsha.io]
>
> *Sent:* Thursday, February 05, 2015 2:35 PM *To:*
> user@storm.apache.org *Subject:* Re: Storm cluster only sees 1 worker
> node at a time - but swtiches between nodes

>


> David,


> It looks like there might be some confusion between supervisor and
> workers.


>


> "Zookeeper is running on separate VMs. 1 VM has the nimbus process,
> and the UI process, and no workers, and the other 2 machines each have
> a worker with 2 ports"


> You have nimbus, UI and two supervisors and each with one worker slot.
> These workers starts running once you deploy a topology.


> Number of workers per supervisor controlled by this config


> supervisor.slots.ports:


> - 6700


> Since you specified only one port than it means each supervisor can
> only spawn one worker when a topology deployed. You can add more ports
> to increase the number of workers per supervisor.


>


> "However when I go to the Storm UI, the cluster summary says there is
> only 1 supervisor with 2 slots, and the supervisor summary on has 1
> row, but the host name switches
 between storm-worker-1 and storm-worker-2. It seems to be seeing both
 machines, but only 1 at a time."


>


> I am assuming you are using same storm.yaml on all the hosts (i.e
> nimbus, supervisors etc) . In this case based on your config each
> supervisor should show it has one worker.


> You should be able to see two supervisor listed under "Supervisor"
> section in storm UI. Do check the logs if your supervisors might be
> missing connection to zookeeper or crashing! .


>


> Which version of storm you are using. It might help if you can attach
> screenshots for storm UI.


> Thanks,


> Harsha


>


>


>


> On Thu, Feb 5, 2015, at 11:05 AM, David Shepherd wrote:


>> I have set up a Storm cluster on 3 vms running on openstack.
>> Zookeeper is running on separate VMs. 1 VM has the nimbus process,
>> and the UI process,
 and no workers, and the other 2 machines each have a worker with 2
 ports. The workers were created by provisioning a VM from an image
 using the openstack tooling, so they are identical.


>>


>> Everything starts up with no errors in any logs I can fin. However
>> when I go to the Storm UI, the cluster summary says there is only 1
>> supervisor
 with 2 slots, and the supervisor summary on has 1 row, but the host
 name switches between storm-worker-1 and storm-worker-2. It seems to be
 seeing both machines, but only 1 at a time.


>>


>> The storm.yaml is:


>>


>> storm.zookeeper.servers:


>> - "zookeeper-host-1"


>> - "zookeeper-host-2"


>> - "zookeeper-host-3"


>> storm.zookeeper.port: 2181


>>


>> nimbus.host: "nimbus"


>> nimbus.thrift.port: 6627


>> storm.local.dir: "/var/opt/stormtmp"


>> java.library.path: "/usr/local/lib"


>>


>> supervisor.slots.ports:


>> - 6700


>> worker.childopts: "-Xmx768m"


>> nimbus.childopts: "-Xmx512m"


>> supervisor.childopts: "-Xmx256m"


>> ui.childopts: "-Xmx512m"


>>


>> I am totally stumped – can’t find anything on this behavior anywhere
>> online or in any of the books I have. If anyone else has experienced
>> this and
 can point me in the right direction it would be much appreciated.


>>


>> Thanks,


>>


>> David


>>


>


> Email had 2 attachments:


>  * image001.png 45k (image/png)
>  * image002.png 46k (image/png)


Mime
View raw message