hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zheng Lv <lvzheng19800...@gmail.com>
Subject Re: is there any problem with our environment?
Date Fri, 09 Oct 2009 10:22:44 GMT
These are the second shut down server's log:

2009-10-09 15:23:12,158 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255072983694 whose
highest sequence/edit id is 8079718
2009-10-09 15:24:28,717 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception  for block
blk_2196913463479121687_23657java.net.SocketTimeoutException: 69000 millis
timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.33.8:33876 remote=/
192.168.33.8:50010]
 at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
 at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
 at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
 at java.io.DataInputStream.readFully(DataInputStream.java:178)
 at java.io.DataInputStream.readLong(DataInputStream.java:399)
 at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2369)
2009-10-09 15:24:34,936 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_2196913463479121687_23657 bad datanode[0]
192.168.33.8:50010
2009-10-09 15:24:34,936 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_2196913463479121687_23657 in pipeline
192.168.33.8:50010, 192.168.33.5:50010, 192.168.33.6:50010: bad datanode
192.168.33.8:50010
2009-10-09 15:24:34,960 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 5 on 60020 took 75039ms appending an edit to hlog;
editcount=6325
2009-10-09 15:24:34,962 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.8:60020.logFlusher took 69944ms optional sync'ing
hlog; editcount=6331
2009-10-09 15:24:35,076 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255072992146,
entries=6415, calcsize=63770126, filesize=63085397. New hlog
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073075063
2009-10-09 15:24:38,543 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255072671531
in 1mins, 44sec
2009-10-09 15:24:38,544 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255072671531
2009-10-09 15:24:39,821 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255072671531
2009-10-09 15:24:41,531 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552/447724835
available; sequence id is 8090065
2009-10-09 15:24:41,531 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552
2009-10-09 15:24:41,690 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552/1620807431
available; sequence id is 8090066
2009-10-09 15:24:41,691 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:24:41,714 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255072671531',
STARTKEY => 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973',
ENDKEY => '', ENCODED => 99294965, OFFLINE => true, SPLIT => true, TABLE =>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552,
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552.
Split took 3sec
2009-10-09 15:24:41,714 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072362203_728621,1255072671531
2009-10-09 15:24:43,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552
2009-10-09 15:24:43,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:24:43,120 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552
2009-10-09 15:24:43,245 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552/447724835
available; sequence id is 8090065
2009-10-09 15:24:43,245 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:24:43,397 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552/1620807431
available; sequence id is 8090066
2009-10-09 15:24:51,275 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073075063,
entries=6410, calcsize=63768309, filesize=63084111. New hlog
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073091269
2009-10-09 15:24:51,276 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255072992146 whose
highest sequence/edit id is 8086133
2009-10-09 15:24:56,814 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 9 on 60020 took 1663ms appending an edit to hlog;
editcount=5165
2009-10-09 15:24:57,227 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072362203_728621,1255072671531
in 15sec
2009-10-09 15:24:57,228 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552
2009-10-09 15:24:58,968 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073091269,
entries=6420, calcsize=63771943, filesize=63086683. New hlog
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073098961
2009-10-09 15:24:58,968 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073075063 whose
highest sequence/edit id is 8092549
2009-10-09 15:24:58,974 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 3 on 60020 took 1149ms appending an edit to hlog; editcount=0
2009-10-09 15:25:03,892 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Too many store files
for region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552:
5, requesting compaction and waiting
2009-10-09 15:25:06,189 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 8 on 60020 took 1077ms appending an edit to hlog;
editcount=6342
2009-10-09 15:25:06,265 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073098961,
entries=6414, calcsize=63770126, filesize=63085212. New hlog
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073106258
2009-10-09 15:25:08,937 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552
in 11sec
2009-10-09 15:25:08,937 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:25:09,897 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Blocking updates for 'IPC Server handler 5 on 60020' on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552:
memstore size 128.0m is >= than blocking 128.0m size
2009-10-09 15:25:25,819 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
in 16sec
2009-10-09 15:25:25,819 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:25:25,900 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Compaction has
completed, we waited 22500ms, finishing flush of region webpage,http:\x2F\
x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:25:29,550 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Unblocking updates for region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
'IPC Server handler 5 on 60020'
2009-10-09 15:25:29,550 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:25:30,065 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073125827/2031468408
available; sequence id is 8110565
2009-10-09 15:25:30,065 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073125827
2009-10-09 15:25:30,708 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072716434_740161,1255073125827/562635413
available; sequence id is 8110566
2009-10-09 15:25:30,708 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072716434_740161,1255073125827
2009-10-09 15:25:30,731 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552',
STARTKEY => 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287',
ENDKEY => '', ENCODED => 1620807431, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073125827,
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072716434_740161,1255073125827.
Split took 4sec
2009-10-09 15:28:05,326 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0001 to sun.nio.ch.SelectionKeyImpl@55229f61
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 15:28:05,406 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0003 to sun.nio.ch.SelectionKeyImpl@540dc4f1
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 15:28:05,426 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-10-09 15:28:05,484 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 15:28:06,195 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 15:28:08,486 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0001 to sun.nio.ch.SelectionKeyImpl@3b4378b0
java.net.NoRouteToHostException: No route to host
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-10-09 15:28:08,486 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:08,487 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:08,487 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0003 to sun.nio.ch.SelectionKeyImpl@4aa88935
java.net.NoRouteToHostException: No route to host
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-10-09 15:28:08,487 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:08,488 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:08,718 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 15:28:09,274 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-10-09 15:28:20,128 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-10-09 15:28:22,058 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0003 to sun.nio.ch.SelectionKeyImpl@7ec81cdd
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 15:28:22,059 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
 at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:22,606 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0001 to sun.nio.ch.SelectionKeyImpl@3df3232
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 15:28:22,606 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
 at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:23,432 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-10-09 15:28:24,563 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 15:28:34,827 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.8:32946 remote=ubuntu10/192.168.33.10:2222]
2009-10-09 15:28:34,827 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-10-09 15:28:34,833 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0003 to sun.nio.ch.SelectionKeyImpl@4ccbf648
java.io.IOException: Session Expired
 at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
 at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-10-09 15:28:34,833 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x24375d149b0003
2009-10-09 15:28:34,834 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x24375d149b0003
2009-10-09 15:28:34,834 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.8:33682 remote=ubuntu12/192.168.33.12:2222]
2009-10-09 15:28:34,834 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x24375d149b0003
2009-10-09 15:28:34,834 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-10-09 15:28:34,835 INFO org.apache.zookeeper.ZooKeeper: Session:
0x24375d149b0003 closed
2009-10-09 15:28:34,835 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-10-09 15:28:34,836 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-10-09 15:28:34,836 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0001 to sun.nio.ch.SelectionKeyImpl@6296185b
java.io.IOException: Session Expired
 at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
 at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-10-09 15:28:34,836 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-10-09 15:28:34,837 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1914)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1899)
 at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 15:28:34,844 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 0)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
 at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
 at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
 at java.lang.Thread.run(Thread.java:619)
2009-10-09 15:28:34,895 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=283, stores=564, storefiles=564, storefileIndexSize=49,
memstoreSize=0, usedHeap=306, maxHeap=2993, blockCacheSize=86974704,
blockCacheFree=540873296, blockCacheCount=693, blockCacheHitRatio=0
2009-10-09 15:28:34,897 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020, call next(-3749056290061789970) from 192.168.33.7:47319:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1914)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1899)
 at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 15:28:34,901 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 1)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
 at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
 at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
 at java.lang.Thread.run(Thread.java:619)
2009-10-09 15:28:34,901 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stop was requested,
clearing the toDo despite of the exception
2009-10-09 15:28:34,902 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-10-09 15:28:34,902 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.close(HRegionServer.java:1946)
 at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 15:28:34,902 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-10-09 15:28:34,903 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-10-09 15:28:34,903 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-10-09 15:28:34,903 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020
2009-10-09 15:28:34,904 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-10-09 15:28:34,904 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-10-09 15:28:34,903 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-10-09 15:28:34,905 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-10-09 15:28:34,905 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-10-09 15:28:34,904 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-10-09 15:28:34,920 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-10-09 15:28:34,920 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2009-10-09 15:28:34,967 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.33.8:60020.logFlusher exiting
2009-10-09 15:28:35,106 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.33.8:60020.cacheFlusher exiting
2009-10-09 15:28:35,110 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2009-10-09 15:28:35,110 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.33.8:60020.compactor exiting
2009-10-09 15:28:35,111 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.33.8:60020.majorCompactionChecker exiting
2009-10-09 15:28:43,136 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.8:60020.leaseChecker closing leases
2009-10-09 15:28:43,136 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.8:60020.leaseChecker closed leases
2009-10-09 15:28:43,402 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-10-09 15:29:34,897 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-10-09 15:29:34,904 WARN org.apache.hadoop.hdfs.DFSClient: Problem
renewing lease for DFSClient_1152927602
java.io.IOException: Call to ubuntu6/192.168.33.7:9000 failed on local
exception: java.io.IOException
 at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
 at org.apache.hadoop.ipc.Client.call(Client.java:742)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
 at $Proxy1.renewLease(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at $Proxy1.renewLease(Unknown Source)
 at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1016)
 at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1028)
 at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException
 at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:416)
 at org.apache.hadoop.ipc.Client$Connection.run(Client.java:445)
Caused by: java.lang.InterruptedException
 ... 2 more

 ... ...

 2009-10-09 15:29:35,245 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.33.8:60020
2009-10-09 15:29:35,246 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x24375d149b0001
2009-10-09 15:29:35,246 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x24375d149b0001
2009-10-09 15:29:35,246 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x24375d149b0001
2009-10-09 15:29:35,246 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-10-09 15:29:35,246 INFO org.apache.zookeeper.ZooKeeper: Session:
0x24375d149b0001 closed
2009-10-09 15:29:35,247 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
192.168.33.8:60020 exiting
2009-10-09 15:29:35,252 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-10-09 15:29:35,252 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message