hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Xing" <xing...@gmail.com>
Subject Re: region server problem
Date Mon, 13 Oct 2008 10:19:31 GMT
It was solved by upping file descriptor number and upgrading hbase version.
The data can be loaded successfully now. Thanks a million for the advices.



But we have observed another weird problem. Nearly all insertion requests
were directed to one region server. So this incurs another question. How
does master do overload balance?



We started 50 insertion clients calling hbase APIs and one table on server
side was created to store data.



Thanks

-Ray

On Mon, Oct 13, 2008 at 12:50 AM, Slava Gorelik <slava.gorelik@gmail.com>wrote:

> In this thread i was asked to provide some information about my hbase
> cluster and some logs. So :
> 1) Hadoop cluster is :
>
> Cluster Summary
>  * * * 4961 files and directories, 2922 blocks = 7883 total. Heap Size is
> 10.38 MB / 888.94 MB (1%)
> *   Capacity : 814.33 GB DFS Remaining : 693.71 GB  DFS Used : 42.46 GB DFS
> Used% : 5.21 %
>
>  Live Datanodes : 7
>
>  Node Last Contact  Admin State Size (GB) Used (%) Used (%) Remaining (GB)
> Blocksredhat010 2In Service 113.434.89 93.981053 redhat011 0In
> Service115.26
> 5.34 97.09 1273 redhat012 1In Service115.265.36 96.95 1162 redhat013 0In
> Service115.264.91 97.69 1291 redhat014 1In Service115.265.48 96.99 1361
> redhat015 2In Service115.265.39 97.12 1291 suse010 1In Service124.625.13
> 113.88 1335
>
> 2) Hbase cluster, the filesize is changed to 64mb and also flushsize to
> 16mb
> (because of small data that is written frequently):
> Master Attributes Attribute NameValueDescription HBase Version0.18.0,
> r697626HBase version and svn revision HBase CompiledSun Sep 21 16:00:50 PDT
> 2008, stackWhen HBase version was compiled and by whom Hadoop
> Version0.18.0,
> r686010Hadoop version and svn revision Hadoop CompiledThu Aug 14 19:48:33
> UTC 2008, hadoopqaWhen Hadoop version was compiled and by whom Filesystem
> hdfs://REDHAT010:9000/hbaseFilesystem HBase is running on HBase Root
> Directoryhdfs://REDHAT010:9000/hbaseLocation of HBase home directory Load
> average43.0Average load across all region servers. Naive computation.
> Catalog
> Tables TableDescription -ROOT-The -ROOT- table holds references to all
> .META. regions. .META.The .META. table holds references to all User Table
> regions User Tables
>
> 1 table(s) in set.
>  TableDescription BizDB {NAME => 'BizDB', IS_ROOT => 'false', IS_META =>
> 'false', FAMILIES => [{NAME => 'BusinessObject', BLOOMFILTER => 'false',
> VERSIONS => '3', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL =>
> '-1',
> IN_MEMORY => 'false', BLOCKCACHE => 'false'}]} Region Servers AddressStart
> CodeLoad redhat011:600201223827465065requests: 0 regions: 43
> redhat012:60020
> 1223827465975requests: 0 regions: 43 redhat013:600201223827465712requests:
> 0
> regions: 43 redhat014:600201223827465249requests: 0 regions: 43
> redhat015.:600201223827465108requests: 0 regions: 43 suse010:60020
> 1223813153133requests: 0 regions: 43 Total: servers: 6 requests: 0 regions:
> 258
>
> 3) Uploader is a simple java program that user BatchUpdate to upload.
> 4) Descriptors are not upped.
> 5) Logs from region server, i found number of exception on the same region
> server:
> 2008-10-08 07:41:58,246 WARN org.apache.hadoop.dfs.DFSClient: Exception
> while reading from blk_2538465098022552520_15050 of
> /hbase/BizDB/486345958/BusinessObject/mapfiles/8802744696946937845/data
> from
> 10.26.237.141:50010: java.io.IOException: Premeture EOF from inputStream
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> at
> org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:996)
> at
>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159)
> at org.apache.hadoop.dfs.DFSClient$BlockReader.read(DFSClient.java:858)
> at
>
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1384)
> at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1420)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at
>
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:64)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:102)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1930)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1830)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:463)
> at org.apache.hadoop.io.MapFile$Reader.getClosest(MapFile.java:558)
> at org.apache.hadoop.io.MapFile$Reader.getClosest(MapFile.java:541)
> at
>
> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.getClosest(HStoreFile.java:761)
> at
>
> org.apache.hadoop.hbase.regionserver.HStore.getFullFromMapFile(HStore.java:1179)
> at org.apache.hadoop.hbase.regionserver.HStore.getFull(HStore.java:1160)
> at org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1221)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1036)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>
> Another exception:
>
> 2008-10-08 08:19:22,218 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> starting compaction on region
> BizDB,1.1.PerfBO1.5eaecb0b-235f-4d62-bae3-f8e42a3f65ee,1223410715671
> 2008-10-08 08:19:22,285 INFO org.apache.hadoop.hbase.regionserver.HLog: New
> log writer created at
> /hbase/log_10.26.237.141_1223394485409_60020/hlog.dat.1223446762266
> 2008-10-08 08:19:22,370 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Could not read from stream
> 2008-10-08 08:19:22,370 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_-2877152584708860910_17060
> 2008-10-08 08:19:22,427 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Could not read from stream
> 2008-10-08 08:19:22,427 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_8480966852058311110_17062
> 2008-10-08 08:19:22,822 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Could not read from stream
> 2008-10-08 08:19:22,822 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_1836763064916871218_17062
> 2008-10-08 08:19:28,402 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.26.237.138:50010
> 2008-10-08 08:19:28,403 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_-6294347938555137047_17063
> 2008-10-08 08:19:28,432 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.26.237.137:50010
> 2008-10-08 08:19:28,432 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_5692207445386295686_17063
> 2008-10-08 08:19:28,828 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.26.237.139:50010
> 2008-10-08 08:19:28,828 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_-5426084204552912284_17063
> 2008-10-08 08:19:34,439 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.26.237.139:50010
> 2008-10-08 08:19:34,440 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_-3084067451611865531_17065
> 2008-10-08 08:19:34,941 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.26.237.140:50010
> 2008-10-08 08:19:34,941 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_8531979798217012059_17068
> 2008-10-08 08:19:40,444 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Could not read from stream
> 2008-10-08 08:19:40,445 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_-690757568573941572_17074
> 2008-10-08 08:19:40,946 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Could not read from stream
> 2008-10-08 08:19:40,946 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_-3282075547420544604_17074
> 2008-10-08 08:19:46,447 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer
> Exception: java.io.IOException: Unable to create new block.
> at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2349)
> at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
> at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)
>
> 2008-10-08 08:19:46,447 WARN org.apache.hadoop.dfs.DFSClient: Error
> Recovery
> for block blk_-690757568573941572_17074 bad datanode[0]
> 2008-10-08 08:19:46,459 ERROR
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction/Split
> failed for region
> BizDB,1.1.PerfBO1.5eaecb0b-235f-4d62-bae3-f8e42a3f65ee,1223410715671
> java.io.IOException: Could not get block locations. Aborting...
> at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
> at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
> at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
> 2008-10-08 08:19:46,461 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> starting compaction on region
> BizDB,1.1.PerfBO1.5c9d1b38-bb27-4693-9466-ded9b4e8c59e,1223412524168
> 2008-10-08 08:19:46,564 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new compressor
> 2008-10-08 08:19:46,569 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Could not read from stream
> 2008-10-08 08:19:46,569 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_-2167445393797967261_17083
> 2008-10-08 08:19:46,951 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Could not read from stream
> 2008-10-08 08:19:46,951 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_5449441848613806871_17083
> 2008-10-08 08:19:52,573 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Could not read from stream
> 2008-10-08 08:19:52,574 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_-7625939221720637541_17092
> 2008-10-08 08:19:52,955 INFO org.apache.hadoop.dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Could not read from stream
> 2008-10-08 08:19:52,955 INFO org.apache.hadoop.dfs.DFSClient: Abandoning
> block blk_-1769229717555876257_17092
> 2008-10-08 08:19:58,957 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer
> Exception: java.io.IOException: Unable to create new block.
> at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2349)
> at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
> at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)
>
> 2008-10-08 08:19:58,958 WARN org.apache.hadoop.dfs.DFSClient: Error
> Recovery
> for block blk_-1769229717555876257_17092 bad datanode[0]
> 2008-10-08 08:19:58,958 FATAL org.apache.hadoop.hbase.regionserver.Flusher:
> Replay of hlog required. Forcing server restart
>
> And another one:
>
> 2008-10-07 22:50:57,896 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting split of region
> BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223403629818
> 2008-10-07 22:50:58,163 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> closed BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223403629818
> 2008-10-07 22:50:58,336 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 6 on 60020, call batchUpdate([B@154c8c3c, row =>
> 1.1.PerfBO1.109900e7-af7b-4bf4-b682-50a46760701c, {column =>
> BusinessObject:s2, value => '...', column => BusinessObject:s1, value =>
> '...', column => BusinessObject:@@identifier@@, value => '...'}, -1) from
> 10.26.237.185:37696: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223403629818 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region
> BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223403629818 closed
> at
>
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-10-07 22:50:58,951 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region
>
> BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223412657926/1465662157
> available
> 2008-10-07 22:50:58,952 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> closed BizDB,1.1.PerfBO1.103c5752-efcd-4510-85eb-d491d5ca1fa9,1223412657926
>
>
> Best Regards.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message