storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vineet Mishra <clearmido...@gmail.com>
Subject [No Subject]
Date Sat, 07 Feb 2015 19:03:13 GMT
Hi All,

I am running a Kafka Storm topology in distributed mode, its running good
for the initial run when I start the cluster(3 node cluster) deploy the
Storm Topology and leave it to run.
There are often times the whole cluster goes down(nimbus, supervisor,
workers) and this is most of the time happening when I submit the topology
to run and disconnect my session from the machine.

I could fairly notice that on one of the worker node its throwing the error
:

java.lang.RuntimeException: java.lang.RuntimeException: Client is being
closed, and does not take requests any more

Config and Detailed Stack Trance is provided below.

Node 1 - Nimbus, UI
Node 2 - Supervisor, Worker
Node 3 - Supervisor, Worker

2015-02-07T23:01:25.884+0530 b.s.d.worker [INFO] Shutting down worker
KafkaConsumerTopologyy-1-1423329275 9d98d0b4-1bb4-42e9-9a72-a67b82c64b2c
6703
2015-02-07T23:01:25.884+0530 b.s.m.n.Client [INFO] Closing Netty Client
Netty-Client-ip-20-0-0-78/20.0.0.78:6703
2015-02-07T23:01:25.885+0530 b.s.m.n.Client [INFO] Waiting for pending
batchs to be sent with Netty-Client-ip-20-0-0-78/20.0.0.78:6703...,
timeout: 600000ms, pendings: 0
2015-02-07T23:01:25.886+0530 b.s.d.worker [INFO] Shutting down receive
thread
2015-02-07T23:01:25.886+0530 o.a.s.c.r.ExponentialBackoffRetry [WARN]
maxRetries too large (300). Pinning to 29
2015-02-07T23:01:25.886+0530 b.s.u.StormBoundedExponentialBackoffRetry
[INFO] The baseSleepTimeMs [100] the maxSleepTimeMs [1000] the maxRetries
[300]
2015-02-07T23:01:25.887+0530 b.s.m.n.Client [INFO] New Netty Client,
connect to localhost, 6703, config: , buffer_size: 5242880
2015-02-07T23:01:25.887+0530 b.s.m.n.Client [INFO] Reconnect started for
Netty-Client-localhost/127.0.0.1:6703... [0]
2015-02-07T23:01:25.887+0530 b.s.m.loader [INFO] Shutting down
receiving-thread: [KafkaConsumerTopologyy-1-1423329275, 6703]
2015-02-07T23:01:25.893+0530 b.s.m.n.Client [INFO] connection established
to a remote host Netty-Client-localhost/127.0.0.1:6703, [id: 0x8f71aaa0, /
127.0.0.1:59427 => localhost/127.0.0.1:6703]
2015-02-07T23:01:25.893+0530 b.s.m.n.Client [INFO] Closing Netty Client
Netty-Client-localhost/127.0.0.1:6703
2015-02-07T23:01:25.893+0530 b.s.m.n.Client [INFO] Waiting for pending
batchs to be sent with Netty-Client-localhost/127.0.0.1:6703..., timeout:
600000ms, pendings: 0
2015-02-07T23:01:25.894+0530 b.s.m.loader [INFO] Waiting for
receiving-thread:[KafkaConsumerTopologyy-1-1423329275, 6703] to die
2015-02-07T23:01:25.895+0530 b.s.m.loader [INFO] Shutdown receiving-thread:
[KafkaConsumerTopologyy-1-1423329275, 6703]
2015-02-07T23:01:25.895+0530 b.s.d.worker [INFO] Shut down receive thread
2015-02-07T23:01:25.895+0530 b.s.d.worker [INFO] Terminating messaging
context
2015-02-07T23:01:25.895+0530 b.s.d.worker [INFO] Shutting down executors
2015-02-07T23:01:25.895+0530 b.s.d.executor [INFO] Shutting down executor
KafkaSpout:[3 3]
2015-02-07T23:01:25.896+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.896+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.897+0530 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: Client is being
closed, and does not take requests any more
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
~[storm-core-0.9.3.jar:0.9.3]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
~[storm-core-0.9.3.jar:0.9.3]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
~[storm-core-0.9.3.jar:0.9.3]
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1460.invoke(disruptor.clj:94)
~[storm-core-0.9.3.jar:0.9.3]
at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463)
~[storm-core-0.9.3.jar:0.9.3]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
Caused by: java.lang.RuntimeException: Client is being closed, and does not
take requests any more
at backtype.storm.messaging.netty.Client.send(Client.java:185)
~[storm-core-0.9.3.jar:0.9.3]
at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54)
~[storm-core-0.9.3.jar:0.9.3]
at
backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__3730$fn__3731.invoke(worker.clj:330)
~[storm-core-0.9.3.jar:0.9.3]
at
backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__3730.invoke(worker.clj:328)
~[storm-core-0.9.3.jar:0.9.3]
at
backtype.storm.disruptor$clojure_handler$reify__1447.onEvent(disruptor.clj:58)
~[storm-core-0.9.3.jar:0.9.3]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.9.3.jar:0.9.3]
... 6 common frames omitted
2015-02-07T23:01:25.900+0530 o.a.z.ZooKeeper [INFO] Session:
0x14b58cc25de0115 closed
2015-02-07T23:01:25.900+0530 o.a.z.ClientCnxn [INFO] EventThread shut down
2015-02-07T23:01:25.900+0530 b.s.d.executor [INFO] Shut down executor
KafkaSpout:[3 3]
2015-02-07T23:01:25.901+0530 b.s.d.executor [INFO] Shutting down executor
KafkaSpout:[5 5]
2015-02-07T23:01:25.901+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.901+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.903+0530 o.a.z.ZooKeeper [INFO] Session:
0x14b58cc25de0117 closed
2015-02-07T23:01:25.903+0530 o.a.z.ClientCnxn [INFO] EventThread shut down
2015-02-07T23:01:25.903+0530 b.s.d.executor [INFO] Shut down executor
KafkaSpout:[5 5]
2015-02-07T23:01:25.903+0530 b.s.d.executor [INFO] Shutting down executor
KafkaSpout:[7 7]
2015-02-07T23:01:25.904+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.904+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.905+0530 o.a.z.ZooKeeper [INFO] Session:
0x14b58cc25de0114 closed
2015-02-07T23:01:25.905+0530 o.a.z.ClientCnxn [INFO] EventThread shut down
2015-02-07T23:01:25.906+0530 b.s.d.executor [INFO] Shut down executor
KafkaSpout:[7 7]
2015-02-07T23:01:25.906+0530 b.s.d.executor [INFO] Shutting down executor
KafkaSpout:[9 9]
2015-02-07T23:01:25.906+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.906+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.906+0530 b.s.util [ERROR] Halting process: ("Async loop
died!")
java.lang.RuntimeException: ("Async loop died!")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325)
[storm-core-0.9.3.jar:0.9.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:92)
[storm-core-0.9.3.jar:0.9.3]
at backtype.storm.util$async_loop$fn__464.invoke(util.clj:473)
[storm-core-0.9.3.jar:0.9.3]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
2015-02-07T23:01:25.908+0530 o.a.z.ZooKeeper [INFO] Session:
0x14b58cc25de0116 closed
2015-02-07T23:01:25.908+0530 o.a.z.ClientCnxn [INFO] EventThread shut down
2015-02-07T23:01:25.908+0530 b.s.d.executor [INFO] Shut down executor
KafkaSpout:[9 9]
2015-02-07T23:01:25.909+0530 b.s.d.executor [INFO] Shutting down executor
__acker:[11 11]
2015-02-07T23:01:25.909+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.909+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.909+0530 b.s.d.executor [INFO] Shut down executor
__acker:[11 11]
2015-02-07T23:01:25.909+0530 b.s.d.executor [INFO] Shutting down executor
__system:[-1 -1]
2015-02-07T23:01:25.910+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.910+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.910+0530 b.s.d.executor [INFO] Shut down executor
__system:[-1 -1]
2015-02-07T23:01:25.910+0530 b.s.d.executor [INFO] Shutting down executor
FileBolt:[1 1]
2015-02-07T23:01:25.910+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.910+0530 b.s.util [INFO] Async loop interrupted!
2015-02-07T23:01:25.911+0530 b.s.d.executor [INFO] Shut down executor
FileBolt:[1 1]
2015-02-07T23:01:25.911+0530 b.s.d.worker [INFO] Shut down executors
2015-02-07T23:01:25.916+0530 b.s.d.worker [INFO] Shutting down transfer
thread

URGENT CALL!

Thanks!

Mime
View raw message