storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Khaled Elmeleegy <kd...@hotmail.com>
Subject workers restarting very shortly after startup
Date Wed, 02 Apr 2014 18:31:34 GMT
Hi,
Workers are restarting very quickly after startup (30s-40s). I am running a topology on my
laptop for testing, so I am guessing tasks are running into some error. The problem is I don't
see any error reported on the log, even though i have the logging level set to debug. Please
see a snippet of the worker log below.
Note that the bolts are reported to be starting, then there is 37s (23:21:31 - 23:22:08) pause
in the log, afterwards the worker seems to restart. Can anyone guess what might be going on
given what's in the log? The second question is there something one can do to make the log
more informative and capture whatever error that might have happened?
Thanks,Khaled



2014-04-01 23:21:30 b.s.d.executor [INFO] Loading executor IngaBolt_01_003:[2 2]2014-04-01
23:21:31 b.s.d.task [INFO] Emitting: IngaBolt_01_003 __system ["startup"]2014-04-01 23:21:31
b.s.d.executor [INFO] Loaded executor tasks IngaBolt_01_003:[2 2]2014-04-01 23:21:31 b.s.d.executor
[INFO] Finished loading executor IngaBolt_01_003:[2 2]2014-04-01 23:21:31 o.a.z.ClientCnxn
[DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null
finished:false header:: 20,3  replyHeader:: 20,74078,0  request:: '/storm/assignments,F  response::
s{47,47,1395528580739,1395528580739,0,61,0,0,1,1,74071} 2014-04-01 23:21:31 o.a.z.ClientCnxn
[DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null
finished:false header:: 21,3  replyHeader:: 21,74078,0  request:: '/storm/storms,F  response::
s{48,48,1395528580741,1395528580741,0,61,0,0,1,1,74070} 2014-04-01 23:21:31 b.s.d.executor
[INFO] Preparing bolt IngaBolt_01_003:(2)2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG] Reading
reply sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null finished:false
header:: 22,3  replyHeader:: 22,74078,0  request:: '/storm/supervisors,F  response:: s{49,49,1395528580743,1395528580743,0,107,0,0,1,1,74061}
2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 23,3  replyHeader:: 23,74078,0  request::
'/storm/workerbeats,F  response:: s{50,50,1395528580745,1395528580745,0,85,0,0,1,1,74069}
2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 24,3  replyHeader:: 24,74078,0  request::
'/storm/errors,F  response:: s{51,51,1395528580747,1395528580747,0,26,0,0,1,0,70999} 2014-04-01
23:21:31 b.s.d.executor [INFO] Loading executor IngaBolt_01_048:[3 3]2014-04-01 23:21:31 b.s.d.task
[INFO] Emitting: IngaBolt_01_048 __system ["startup"]2014-04-01 23:21:31 b.s.d.executor [INFO]
Loaded executor tasks IngaBolt_01_048:[3 3]2014-04-01 23:21:31 b.s.d.executor [INFO] Finished
loading executor IngaBolt_01_048:[3 3]2014-04-01 23:21:31 b.s.d.executor [INFO] Preparing
bolt IngaBolt_01_048:(3)2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986,
packet:: clientPath:null serverPath:null finished:false header:: 25,3  replyHeader:: 25,74078,0
 request:: '/storm/assignments,F  response:: s{47,47,1395528580739,1395528580739,0,61,0,0,1,1,74071}
2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet::
clientPath:null serverPath:null finished:false header:: 26,3  replyHeader:: 26,74078,0  request::
'/storm/storms,F  response:: s{48,48,1395528580741,1395528580741,0,61,0,0,1,1,74070} 2014-04-01
23:21:31 o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null
serverPath:null finished:false header:: 27,3  replyHeader:: 27,74078,0  request:: '/storm/supervisors,F
 response:: s{49,49,1395528580743,1395528580743,0,107,0,0,1,1,74061} 2014-04-01 23:21:31 o.a.z.ClientCnxn
[DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null
finished:false header:: 28,3  replyHeader:: 28,74078,0  request:: '/storm/workerbeats,F  response::
s{50,50,1395528580745,1395528580745,0,85,0,0,1,1,74069} 2014-04-01 23:21:31 o.a.z.ClientCnxn
[DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null
finished:false header:: 29,3  replyHeader:: 29,74078,0  request:: '/storm/errors,F  response::
s{51,51,1395528580747,1395528580747,0,26,0,0,1,0,70999} 2014-04-01 23:21:31 b.s.d.executor
[INFO] Loading executor IngaBolt_01_048:[4 4]2014-04-01 23:21:31 b.s.d.task [INFO] Emitting:
IngaBolt_01_048 __system ["startup"]2014-04-01 23:21:31 b.s.d.executor [INFO] Loaded executor
tasks IngaBolt_01_048:[4 4]2014-04-01 23:21:31 b.s.d.executor [INFO] Finished loading executor
IngaBolt_01_048:[4 4]2014-04-01 23:21:31 b.s.d.executor [INFO] Preparing bolt IngaBolt_01_048:(4)2014-04-01
23:21:31 o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null
serverPath:null finished:false header:: 30,3  replyHeader:: 30,74078,0  request:: '/storm/assignments,F
 response:: s{47,47,1395528580739,1395528580739,0,61,0,0,1,1,74071} 2014-04-01 23:21:31 o.a.z.ClientCnxn
[DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null
finished:false header:: 31,3  replyHeader:: 31,74078,0  request:: '/storm/storms,F  response::
s{48,48,1395528580741,1395528580741,0,61,0,0,1,1,74070} 2014-04-01 23:21:31 o.a.z.ClientCnxn
[DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null
finished:false header:: 32,3  replyHeader:: 32,74078,0  request:: '/storm/supervisors,F  response::
s{49,49,1395528580743,1395528580743,0,107,0,0,1,1,74061} 2014-04-01 23:21:31 o.a.z.ClientCnxn
[DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null
finished:false header:: 33,3  replyHeader:: 33,74078,0  request:: '/storm/workerbeats,F  response::
s{50,50,1395528580745,1395528580745,0,85,0,0,1,1,74069} 2014-04-01 23:21:31 o.a.z.ClientCnxn
[DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null
finished:false header:: 34,3  replyHeader:: 34,74078,0  request:: '/storm/errors,F  response::
s{51,51,1395528580747,1395528580747,0,26,0,0,1,0,70999} 2014-04-01 23:22:08 o.a.z.ZooKeeper
[INFO] Client environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT2014-04-01
23:22:08 o.a.z.ZooKeeper [INFO] Client environment:host.name=172.16.114.112014-04-01 23:22:08
o.a.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_512014-04-01 23:22:08 o.a.z.ZooKeeper
[INFO] Client environment:java.vendor=Oracle Corporation2014-04-01 23:22:08 o.a.z.ZooKeeper
[INFO] Client environment:java.class.path=/Users/kelmeleegy/install/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/storm-core-0.9.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/asm-4.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/clojure-1.4.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/clout-1.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-codec-1.4.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-exec-1.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-io-1.4.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-lang-2.5.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/compojure-1.1.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/core.incubator-0.1.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/guava-13.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/httpcore-4.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/jetty-6.1.26.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/jline-0.9.94.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/joda-time-2.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/json-simple-1.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/junit-3.8.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/kryo-2.17.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/math.numeric-tower-0.0.1.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/minlog-1.2.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/objenesis-1.2.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/servlet-api-2.5-20081211.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/servlet-api-2.5.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/Users/kelmeleegy/install/storm-0.9.0.1/conf:storm-local/supervisor/stormdist/test-1-1396419679/stormjar.jar
 		 	   		  
Mime
View raw message