hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zheng Lv <lvzheng19800...@gmail.com>
Subject Re: is there any problem with our environment?
Date Fri, 09 Oct 2009 10:20:35 GMT
These are the first shut down server's logs:

2009-10-09 13:55:34,708 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@73d1f6da
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 13:55:34,720 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@240ede73
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 13:55:34,821 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-10-09 13:55:34,897 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-10-09 13:55:35,510 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 13:55:38,518 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@54547a3a
java.net.NoRouteToHostException: No route to host
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-10-09 13:55:38,519 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 13:55:38,519 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 13:55:38,707 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 13:55:46,589 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-10-09 13:55:48,228 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@25cd7751
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 13:55:48,229 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
 at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 13:55:48,487 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 13:57:20,244 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255067890842_526799,1255068212004
2009-10-09 13:57:26,146 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 8 on 60020 took 1714ms appending an edit to hlog;
editcount=1537
... ...
2009-10-09 14:00:55,549 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068061432_537933,1255068399681',
STARTKEY => 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068061432_537933',
ENDKEY => '', ENCODED => 1740841210, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068061432_537933,1255068399682,
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068126127_540247,1255068399682.
Split took 3sec
2009-10-09 14:01:22,998 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@1ce8f17d
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 14:01:23,099 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-10-09 14:01:24,067 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 14:01:36,328 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@7624a116
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 14:01:37,317 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-10-09 14:01:37,398 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@a5612e0
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 14:01:37,398 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
 at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:38,042 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-10-09 14:01:40,318 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@18f1356d
java.net.NoRouteToHostException: No route to host
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-10-09 14:01:40,319 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:40,319 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:40,320 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@3ea93e6
java.net.NoRouteToHostException: No route to host
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-10-09 14:01:40,320 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:40,320 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:40,621 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-10-09 14:01:41,404 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 14:01:42,402 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 14:01:54,738 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@581f37ff
java.io.IOException: TIMED OUT
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 14:01:54,739 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
 at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:55,499 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:38517 remote=ubuntu2/192.168.33.9:2222]
2009-10-09 14:01:55,500 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-10-09 14:01:55,507 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@6995738b
java.io.IOException: Connection reset by peer
 at sun.nio.ch.FileDispatcher.read0(Native Method)
 at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
 at sun.nio.ch.IOUtil.read(IOUtil.java:200)
 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
 at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:651)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-10-09 14:01:55,507 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
 at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
 at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:55,508 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
 at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:55,712 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 14:01:55,713 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:57273 remote=ubuntu12/192.168.33.12:2222]
2009-10-09 14:01:55,713 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-10-09 14:01:55,715 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-10-09 14:01:55,715 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@15f4a8c8
java.io.IOException: Session Expired
 at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
 at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-10-09 14:01:55,715 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-10-09 14:01:55,803 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=168, stores=335, storefiles=337, storefileIndexSize=28,
memstoreSize=0, usedHeap=237, maxHeap=2993, blockCacheSize=5151568,
blockCacheFree=622696432, blockCacheCount=0, blockCacheHitRatio=0
2009-10-09 14:01:56,156 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 14:01:56,156 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:57274 remote=ubuntu12/192.168.33.12:2222]
2009-10-09 14:01:56,156 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-10-09 14:01:56,507 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-10-09 14:01:58,861 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-10-09 14:02:00,399 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.33.5:60020.logFlusher exiting
2009-10-09 14:02:01,277 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2009-10-09 14:02:02,713 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.33.5:60020.cacheFlusher exiting
2009-10-09 14:02:06,216 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020, call getClosestRowBefore([B@3925421d, [B@3d179b3a,
[B@58d01c2b) from 192.168.33.7:46653: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
 at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:07,771 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020, call put([B@9ecead5,
[Lorg.apache.hadoop.hbase.client.Put;@212d4b0a) from 192.168.33.6:52759:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:08,228 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020, call getClosestRowBefore([B@24998e8a, [B@5d5d18cb,
[B@b06e750) from 192.168.33.7:46653: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
 at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:09,850 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020, call put([B@723c3ad7,
[Lorg.apache.hadoop.hbase.client.Put;@503b33d8) from 192.168.33.6:52759:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:10,233 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020, call getClosestRowBefore([B@5ec8d30e, [B@52986cbf,
[B@150b99f6) from 192.168.33.7:46653: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
 at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:11,856 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020, call put([B@5a8b6e61,
[Lorg.apache.hadoop.hbase.client.Put;@77fbd0b7) from 192.168.33.6:52759:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:12,241 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020, call getClosestRowBefore([B@3543dbeb, [B@7a1b3c9e,
[B@1228e7b3) from 192.168.33.7:46653: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
 at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:15,550 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.33.5:60020.compactor exiting
2009-10-09 14:02:16,246 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020, call getClosestRowBefore([B@3f062f39, [B@2f066782,
[B@4d759fec) from 192.168.33.7:46653: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
 at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:18,070 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020, call put([B@b6f5d7c,
[Lorg.apache.hadoop.hbase.client.Put;@3c627dc1) from 192.168.33.7:46658:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:33,974 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020, call put([B@2be39fa6,
[Lorg.apache.hadoop.hbase.client.Put;@6d0661cc) from 192.168.33.8:33100:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:03:18,164 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 0)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
 at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
 at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
 at java.lang.Thread.run(Thread.java:619)
2009-10-09 14:03:18,165 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stop was requested,
clearing the toDo despite of the exception
2009-10-09 14:03:18,165 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-10-09 14:03:18,166 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-10-09 14:03:18,166 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-10-09 14:03:18,173 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020
2009-10-09 14:03:18,173 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-10-09 14:03:18,174 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2009-10-09 14:03:18,173 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2009-10-09 14:03:18,173 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-10-09 14:03:18,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-10-09 14:03:18,176 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-10-09 14:03:18,176 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-10-09 14:03:18,174 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-10-09 14:03:18,174 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-10-09 14:03:18,177 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-10-09 14:03:18,430 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.33.5:60020.majorCompactionChecker exiting
2009-10-09 14:03:24,535 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.5:60020.leaseChecker closing leases
2009-10-09 14:03:24,535 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.5:60020.leaseChecker closed leases
2009-10-09 14:04:09,451 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception  for block
blk_1789127546305167321_21177java.io.IOException: Bad response 1 for block
blk_1789127546305167321_21177 from datanode 192.168.33.8:50010
 at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2393)
2009-10-09 14:04:09,452 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:09,452 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:09,532 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.5:50010 failed 1 times.  Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:10,532 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:10,532 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:10,564 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.5:50010 failed 2 times.  Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:11,564 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:11,564 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:11,573 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.5:50010 failed 3 times.  Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:12,574 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:12,574 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:12,590 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.5:50010 failed 4 times.  Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:13,590 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:13,591 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:13,603 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.5:50010 failed 5 times.  Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:14,603 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:14,603 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:16,242 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.5:50010 failed 6 times.  Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Marking primary
datanode as bad.
2009-10-09 14:04:17,310 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.6:50010 failed 1 times.  Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:18,641 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.6:50010 failed 2 times.  Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:19,654 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.6:50010 failed 3 times.  Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:20,668 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.6:50010 failed 4 times.  Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:21,684 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.6:50010 failed 5 times.  Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:22,708 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.6:50010 failed 6 times.  Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Marking primary datanode as bad.
2009-10-09 14:04:23,755 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.8:50010 failed 1 times.  Pipeline was
192.168.33.8:50010. Will retry...
2009-10-09 14:04:24,765 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.8:50010 failed 2 times.  Pipeline was
192.168.33.8:50010. Will retry...
2009-10-09 14:04:25,773 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.8:50010 failed 3 times.  Pipeline was
192.168.33.8:50010. Will retry...
2009-10-09 14:04:26,783 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.8:50010 failed 4 times.  Pipeline was
192.168.33.8:50010. Will retry...
2009-10-09 14:04:27,792 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.8:50010 failed 5 times.  Pipeline was
192.168.33.8:50010. Will retry...
2009-10-09 14:04:28,800 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed  because recovery
from primary datanode 192.168.33.8:50010 failed 6 times.  Pipeline was
192.168.33.8:50010. Aborting...
2009-10-09 14:04:28,800 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
abort
java.io.IOException: Error Recovery for block blk_1789127546305167321_21177
failed  because recovery from primary datanode 192.168.33.8:50010 failed 6
times.  Pipeline was 192.168.33.8:50010. Aborting...
 at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2554)
 at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2048)
 at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2211)
2009-10-09 14:04:28,802 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255067721595_511167,1255067964885
2009-10-09 14:04:28,802 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255061989830_96119,1255062349801
... ...
2009-10-09 14:04:28,854 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254221224742_62772,1254221761034
2009-10-09 14:04:28,854 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255067666341_505743,1255067964883
2009-10-09 14:04:28,854 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.33.5:60020
2009-10-09 14:04:28,854 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x124375a36140000
2009-10-09 14:04:28,854 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x124375a36140000
2009-10-09 14:04:28,855 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x124375a36140000
2009-10-09 14:04:28,855 INFO org.apache.zookeeper.ZooKeeper: Session:
0x124375a36140000 closed
2009-10-09 14:04:28,855 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-10-09 14:04:28,899 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
192.168.33.5:60020 exiting
2009-10-09 14:04:28,909 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-10-09 14:04:28,909 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message