hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karthik Manimaran <karthikmanima...@gmail.com>
Subject HBase startup issues
Date Mon, 07 Feb 2011 20:58:39 GMT
Hi,

My HBase setup was running fine for a couple of months and all of a sudden
the following issues has cropped up. The master will shutdown immediately
after startup. The Hadoop datanode is running fine and hdfs status is
Healthy. Any ideas on what could be happening and steps on how to fix/bypass
these errors and get HBase running again? Here are the relevant errors from
logs:

2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient:
DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File
/hbase/.META./1028785192/info/89629418893503418 could only be
replicated to 0 nodes, instead of 1
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

	at org.apache.hadoop.ipc.Client.call(Client.java:739)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
	at $Proxy0.addBlock(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
	at $Proxy0.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)

2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block null bad datanode[0] nodes == null
2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient: Could
not get block locations. Source file
"/hbase/.META./1028785192/info/89629418893503418" - Aborting...
2011-02-07 14:39:45,644 FATAL
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog
required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region: .META.,,1
	at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:946)
	at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:839)
	at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
	at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
File /hbase/.META./1028785192/info/89629418893503418 could only be
replicated to 0 nodes, instead of 1
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

	at org.apache.hadoop.ipc.Client.call(Client.java:739)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
	at $Proxy0.addBlock(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
	at $Proxy0.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
2011-02-07 14:39:45,662 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=73, stores=227, storefiles=184,
storefileIndexSize=1, memstoreSize=0, usedHeap=37, maxHeap=1993,
blockCacheSize=1962504, blockCacheFree=416130968, blockCacheCount=1,
blockCacheHitRatio=75
2011-02-07 14:39:45,662 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
RegionServer:0.cacheFlusher exiting
2011-02-07 14:39:48,481 INFO org.apache.hadoop.ipc.HBaseServer:
Stopping server on 37024
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 3 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer:
Stopping IPC Server listener on 37024
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 0 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 1 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 2 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 5 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 7 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 9 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 4 on 37024: exiting
2011-02-07 14:39:48,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 6 on 37024: exiting
2011-02-07 14:39:48,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 8 on 37024: exiting
2011-02-07 14:39:48,488 INFO org.apache.hadoop.ipc.HBaseServer:
Stopping IPC Server Responder
2011-02-07 14:39:48,488 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping
infoServer
2011-02-07 14:39:48,490 DEBUG
org.apache.hadoop.hbase.regionserver.HLog: closing hlog writer in
hdfs://ec2-75-101-139-144.compute-1.amazonaws.com:9000/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809
2011-02-07 14:39:48,490 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
RegionServer:0.logFlusher exiting
2011-02-07 14:39:48,490 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
RegionServer:0.compactor exiting
2011-02-07 14:39:48,490 INFO
org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
2011-02-07 14:39:48,490 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
RegionServer:0.majorCompactionChecker exiting
2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient:
DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File
/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949
could only be replicated to 0 nodes, instead of 1
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

	at org.apache.hadoop.ipc.Client.call(Client.java:739)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
	at $Proxy0.addBlock(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
	at $Proxy0.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)

2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block null bad datanode[0] nodes == null
2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient: Could
not get block locations. Source file
"/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949"
- Aborting...
2011-02-07 14:39:48,501 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close
log in abort
java.io.IOException: java.io.IOException: File
/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949
could only be replicated to 0 nodes, instead of 1
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
	at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631)
	at java.lang.Thread.run(Thread.java:619)
2011-02-07 14:39:48,502 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
user_activity,,1288475114682
2011-02-07 14:39:48,502 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Closing
user_activity,,1288475114682: compactions & flushes disabled
2011-02-07 14:39:48,502 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for
region, no outstanding scanners on user_activity,,1288475114682
2011-02-07 14:39:48,502 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
outstanding on region user_activity,,1288475114682
2011-02-07 14:39:48,503 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed comments
2011-02-07 14:39:48,503 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed company_ratings
2011-02-07 14:39:48,504 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed details
2011-02-07 14:39:48,504 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed event
2011-02-07 14:39:48,505 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed event_details
2011-02-07 14:39:48,505 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed likes
2011-02-07 14:39:48,505 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed tags
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed user_ratings
2011-02-07 14:39:48,506 INFO
org.apache.hadoop.hbase.regionserver.HRegion: Closed
user_activity,,1288475114682
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
company_analytics_post_a_job_tab,,1288475078918
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Closing
company_analytics_post_a_job_tab,,1288475078918: compactions & flushes
disabled
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for
region, no outstanding scanners on
company_analytics_post_a_job_tab,,1288475078918
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
outstanding on region company_analytics_post_a_job_tab,,1288475078918
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed system_details
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed viewer_details


Thanks,
Karthik Manimaran.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message