hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Slava Gorelik" <slava.gore...@gmail.com>
Subject Re: region server problem
Date Sun, 12 Oct 2008 16:50:22 GMT
In this thread i was asked to provide some information about my hbase
cluster and some logs. So :
1) Hadoop cluster is :

Cluster Summary
 * * * 4961 files and directories, 2922 blocks = 7883 total. Heap Size is
10.38 MB / 888.94 MB (1%)
*   Capacity : 814.33 GB DFS Remaining : 693.71 GB  DFS Used : 42.46 GB DFS
Used% : 5.21 %

 Live Datanodes : 7

  Node Last Contact  Admin State Size (GB) Used (%) Used (%) Remaining (GB)
Blocksredhat010 2In Service 113.434.89 93.981053 redhat011 0In Service115.26
5.34 97.09 1273 redhat012 1In Service115.265.36 96.95 1162 redhat013 0In
Service115.264.91 97.69 1291 redhat014 1In Service115.265.48 96.99 1361
redhat015 2In Service115.265.39 97.12 1291 suse010 1In Service124.625.13
113.88 1335

2) Hbase cluster, the filesize is changed to 64mb and also flushsize to 16mb
(because of small data that is written frequently):
Master Attributes Attribute NameValueDescription HBase Version0.18.0,
r697626HBase version and svn revision HBase CompiledSun Sep 21 16:00:50 PDT
2008, stackWhen HBase version was compiled and by whom Hadoop Version0.18.0,
r686010Hadoop version and svn revision Hadoop CompiledThu Aug 14 19:48:33
UTC 2008, hadoopqaWhen Hadoop version was compiled and by whom Filesystem
hdfs://REDHAT010:9000/hbaseFilesystem HBase is running on HBase Root
Directoryhdfs://REDHAT010:9000/hbaseLocation of HBase home directory Load
average43.0Average load across all region servers. Naive computation. Catalog
Tables TableDescription -ROOT-The -ROOT- table holds references to all
.META. regions. .META.The .META. table holds references to all User Table
regions User Tables

1 table(s) in set.
 TableDescription BizDB {NAME => 'BizDB', IS_ROOT => 'false', IS_META =>
'false', FAMILIES => [{NAME => 'BusinessObject', BLOOMFILTER => 'false',
VERSIONS => '3', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1',
IN_MEMORY => 'false', BLOCKCACHE => 'false'}]} Region Servers AddressStart
CodeLoad redhat011:600201223827465065requests: 0 regions: 43 redhat012:60020
1223827465975requests: 0 regions: 43 redhat013:600201223827465712requests: 0
regions: 43 redhat014:600201223827465249requests: 0 regions: 43
redhat015.:600201223827465108requests: 0 regions: 43 suse010:60020
1223813153133requests: 0 regions: 43 Total: servers: 6 requests: 0 regions:
258

3) Uploader is a simple java program that user BatchUpdate to upload.
4) Descriptors are not upped.
5) Logs from region server, i found number of exception on the same region
server:
2008-10-08 07:41:58,246 WARN org.apache.hadoop.dfs.DFSClient: Exception
while reading from blk_2538465098022552520_15050 of
/hbase/BizDB/486345958/BusinessObject/mapfiles/8802744696946937845/data from
10.26.237.141:50010: java.io.IOException: Premeture EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:996)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236)
at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191)
at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159)
at org.apache.hadoop.dfs.DFSClient$BlockReader.read(DFSClient.java:858)
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1384)
at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1420)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:64)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:102)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1930)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1830)
at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:463)
at org.apache.hadoop.io.MapFile$Reader.getClosest(MapFile.java:558)
at org.apache.hadoop.io.MapFile$Reader.getClosest(MapFile.java:541)
at
org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.getClosest(HStoreFile.java:761)
at
org.apache.hadoop.hbase.regionserver.HStore.getFullFromMapFile(HStore.java:1179)
at org.apache.hadoop.hbase.regionserver.HStore.getFull(HStore.java:1160)
at org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1221)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1036)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Another exception:

2008-10-08 08:19:22,218 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region
BizDB,1.1.PerfBO1.5eaecb0b-235f-4d62-bae3-f8e42a3f65ee,1223410715671
2008-10-08 08:19:22,285 INFO org.apache.hadoop.hbase.regionserver.HLog: New
log writer created at
/hbase/log_10.26.237.141_1223394485409_60020/hlog.dat.1223446762266
2008-10-08 08:19:22,370 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Could not read from stream
2008-10-08 08:19:22,370 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_-2877152584708860910_17060
2008-10-08 08:19:22,427 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Could not read from stream
2008-10-08 08:19:22,427 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_8480966852058311110_17062
2008-10-08 08:19:22,822 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Could not read from stream
2008-10-08 08:19:22,822 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_1836763064916871218_17062
2008-10-08 08:19:28,402 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.26.237.138:50010
2008-10-08 08:19:28,403 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_-6294347938555137047_17063
2008-10-08 08:19:28,432 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.26.237.137:50010
2008-10-08 08:19:28,432 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_5692207445386295686_17063
2008-10-08 08:19:28,828 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.26.237.139:50010
2008-10-08 08:19:28,828 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_-5426084204552912284_17063
2008-10-08 08:19:34,439 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.26.237.139:50010
2008-10-08 08:19:34,440 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_-3084067451611865531_17065
2008-10-08 08:19:34,941 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.26.237.140:50010
2008-10-08 08:19:34,941 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_8531979798217012059_17068
2008-10-08 08:19:40,444 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Could not read from stream
2008-10-08 08:19:40,445 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_-690757568573941572_17074
2008-10-08 08:19:40,946 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Could not read from stream
2008-10-08 08:19:40,946 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_-3282075547420544604_17074
2008-10-08 08:19:46,447 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer
Exception: java.io.IOException: Unable to create new block.
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2349)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)

2008-10-08 08:19:46,447 WARN org.apache.hadoop.dfs.DFSClient: Error Recovery
for block blk_-690757568573941572_17074 bad datanode[0]
2008-10-08 08:19:46,459 ERROR
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction/Split
failed for region
BizDB,1.1.PerfBO1.5eaecb0b-235f-4d62-bae3-f8e42a3f65ee,1223410715671
java.io.IOException: Could not get block locations. Aborting...
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
2008-10-08 08:19:46,461 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region
BizDB,1.1.PerfBO1.5c9d1b38-bb27-4693-9466-ded9b4e8c59e,1223412524168
2008-10-08 08:19:46,564 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new compressor
2008-10-08 08:19:46,569 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Could not read from stream
2008-10-08 08:19:46,569 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_-2167445393797967261_17083
2008-10-08 08:19:46,951 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Could not read from stream
2008-10-08 08:19:46,951 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_5449441848613806871_17083
2008-10-08 08:19:52,573 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Could not read from stream
2008-10-08 08:19:52,574 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_-7625939221720637541_17092
2008-10-08 08:19:52,955 INFO org.apache.hadoop.dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Could not read from stream
2008-10-08 08:19:52,955 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
block blk_-1769229717555876257_17092
2008-10-08 08:19:58,957 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer
Exception: java.io.IOException: Unable to create new block.
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2349)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)

2008-10-08 08:19:58,958 WARN org.apache.hadoop.dfs.DFSClient: Error Recovery
for block blk_-1769229717555876257_17092 bad datanode[0]
2008-10-08 08:19:58,958 FATAL org.apache.hadoop.hbase.regionserver.Flusher:
Replay of hlog required. Forcing server restart

And another one:

2008-10-07 22:50:57,896 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region
BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223403629818
2008-10-07 22:50:58,163 INFO org.apache.hadoop.hbase.regionserver.HRegion:
closed BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223403629818
2008-10-07 22:50:58,336 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 6 on 60020, call batchUpdate([B@154c8c3c, row =>
1.1.PerfBO1.109900e7-af7b-4bf4-b682-50a46760701c, {column =>
BusinessObject:s2, value => '...', column => BusinessObject:s1, value =>
'...', column => BusinessObject:@@identifier@@, value => '...'}, -1) from
10.26.237.185:37696: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223403629818 closed
org.apache.hadoop.hbase.NotServingRegionException: Region
BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223403629818 closed
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-10-07 22:50:58,951 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region
BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223412657926/1465662157
available
2008-10-07 22:50:58,952 INFO org.apache.hadoop.hbase.regionserver.HRegion:
closed BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223412657926


Best Regards.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message