hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 陈加俊 <cjjvict...@gmail.com>
Subject Regionserver is crashed frequently these days
Date Fri, 01 Apr 2011 16:01:32 GMT
Regionserver is crashed frequently  these days.,but It worked fine many
months before these days .
Some logs of one RS's log is as follows:

2011-04-01 19:13:40,413 WARN org.apache.hadoop.hbase.regionserver.Store:
Failed open of hdfs://
master.uc.uuwatch.com:9000/hbase/cjjHTML/1494733632/page/5173469199902346167.1864097884;
presumption is that file was corrupted at flush and lost edits picked up by
commit log replay. Verify!
java.io.IOException: Cannot open filename
/hbase/cjjHTML/1864097884/page/5173469199902346167
......

2011-04-01 19:17:22,716 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
cjjHTML,http://news.ifeng.com/gundong/detail_2011_03/15/515
4913_0.shtml,1300245193111: Overloaded
2011-04-01 19:17:22,716 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_CLOSE: cjjHTML,http://news.ifeng.com/gundong/detail_2011_0
3/15/5154913_0.shtml,1300245193111: Overloaded
2011-04-01 19:17:22,716 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed cjjHTML,
http://news.ifeng.com/gundong/detail_2011_03/15/5154913_0.shtml,1300245193111
2011-04-01 19:17:22,716 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
cjjHTML,http://news.39.net/kyfx/2010124/1561005.html,1298035330808:
Overloaded
2011-04-01 19:17:22,716 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_CLOSE: cjjHTML,
http://news.39.net/kyfx/2010124/1561005.html,1298035330808: Overloaded
......

2011-04-01 22:01:49,212 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x942f0f7ae13d0000 to sun.nio.ch.SelectionKeyImpl@34819c89
java.io.IOException: TIMED OUT
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2011-04-01 22:01:49,213 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x982f0f7ae0960001 to sun.nio.ch.SelectionKeyImpl@7e9ca589
java.io.IOException: TIMED OUT
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2011-04-01 22:01:49,310 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
        at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
        at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:549)
        at java.lang.Thread.run(Thread.java:619)
2011-04-01 22:01:49,313 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2011-04-01 22:01:49,589 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server /192.168.5.155:2181
2011-04-01 22:01:49,589 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.5.149:33651 remote=/192.168.5.155:2181]
2011-04-01 22:01:49,590 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2011-04-01 22:01:49,592 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x982f0f7ae0960001 to sun.nio.ch.SelectionKeyImpl@a5858e9
java.io.IOException: Session Expired
        at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
        at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2011-04-01 22:01:49,592 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2011-04-01 22:01:49,592 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2011-04-01 22:01:49,602 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=941, stores=1721, storefiles=1439,
storefileIndexSize=228, memstoreSize=558, compactionQueueSize=12,
usedHeap=2896, maxHeap=3995, blockCacheSize=640319368,
blockCacheFree=197676344, blockCacheCount=8309, blockCacheHitRatio=93,
fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
2011-04-01 22:01:49,853 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server /192.168.5.147:2181
2011-04-01 22:01:49,854 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.5.149:50014 remote=/192.168.5.147:2181]
2011-04-01 22:01:49,854 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2011-04-01 22:01:49,856 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x942f0f7ae13d0000 to sun.nio.ch.SelectionKeyImpl@35c591b7
java.io.IOException: Session Expired
        at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
        at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2011-04-01 22:01:49,856 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x942f0f7ae13d0000
2011-04-01 22:01:49,857 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x942f0f7ae13d0000
2011-04-01 22:01:49,857 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x942f0f7ae13d0000
2011-04-01 22:01:49,857 INFO org.apache.zookeeper.ZooKeeper: Session:
0x942f0f7ae13d0000 closed
2011-04-01 22:01:49,857 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2011-04-01 22:01:50,310 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2011-04-01 22:01:50,311 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2011-04-01 22:01:50,311 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
......



-- 
Thanks & Best regards
jiajun

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message