hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From charan kumar <charan.ku...@gmail.com>
Subject Region server shutdown during writes (bad data nodes)
Date Sat, 05 Feb 2011 00:15:29 GMT
Hello,

   We are running into a Region server shutdown again during write loads (90
clients) , with Connection rest by peer issue? Any suggestions..

  Setup: 30 Nodes. Hbase 0.90.0, Hadoop-append , CentOS, dell 1950 6G RAM.

  2011-02-04 02:36:16,808 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception  for block
blk_-4303650603271778933_2022254java.io.IOException: Connection reset by
peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
        at sun.nio.ch.IOUtil.read(IOUtil.java:206)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
        at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
        at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
        at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
        at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at java.io.DataInputStream.readLong(DataInputStream.java:399)
        at
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:122)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2547)

2011-02-04 02:36:16,809 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-4303650603271778933_2022254 bad datanode[0]
10.76.99.115:50010
2011-02-04 02:36:16,880 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
serverName=nafhdi12708mwh.io.askjeeves.info,60020,1296782105691,
load=(requests=63, regions=165, usedHeap=2274, maxHeap=4070): Failed open of
daughter
compresstable,\x074k\xB6\x91\xC6\x98\x87,1296815758006.86cf8a61169de38e7ea72fb01c351eb1.
java.io.IOException: All datanodes XXXXXXXXXXXX:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2680)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371)

hdfs config:

        <property>
          <name>dfs.datanode.max.xcievers</name>
          <value>4096</value>
        </property>

        <property>
                <name>dfs.replication</name>
                 <value>3</value>
        </property>

        <property>
                <name>dfs.datanode.du.reserved</name>
                <value>5368709120</value>
        </property>

        <property>
                <name>dfs.datanode.handler.count</name>
                <value>100</value>
        </property>
        <property>
                <name>dfs.namenode.handler.count</name>
                <value>100</value>
        </property>

        <property>
                <name>dfs.datanode.socket.write.timeout</name>
                <value>0</value>

GC OPTS: -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xmn256m
-XX:CMSInitiatingOccupancyFraction=70

Thanks,
Charan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message