hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Bangzhong <bangzh...@gmail.com>
Subject Re: hbase performance
Date Fri, 02 Apr 2010 08:13:17 GMT
region server shutdown when running on 2 clients. From the log, seems region
server can't connect to datanode.

Here is the region server log. I didn't find any exception in the datanode
log.

 to hlog; editcount=291
2010-04-02 16:00:22,376 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception  for block
blk_5538189142737707248_12736java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readLong(DataInputStream.java:399)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2399)

2010-04-02 16:00:22,376 INFO org.apache.hadoop.hdfs.DFSClient: Could not
obtain block blk_-2614684857813896108_12085 from any node:
java.io.IOException: No live nodes contain current block
2010-04-02 16:00:22,376 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_5538189142737707248_12736 bad datanode[0]
172.31.13.190:50010
2010-04-02 16:00:22,377 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
        at sun.nio.ch.IOUtil.write(IOUtil.java:75)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
        at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
        at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
        at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
        at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290)

2010-04-02 16:00:22,377 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception  for block
blk_1462025791805252013_12727java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readLong(DataInputStream.java:399)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2399)

2010-04-02 16:00:22,378 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1462025791805252013_12727 bad datanode[0]
172.31.13.190:50010
2010-04-02 16:00:22,379 INFO org.apache.hadoop.hdfs.DFSClient: Could not
obtain block blk_-5108099522205897784_11310 from any node:
java.io.IOException: No live nodes contain current block
2010-04-02 16:00:22,378 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-7928413171578270686_12744 bad datanode[0]
172.31.13.190:50010
2010-04-02 16:00:22,381 FATAL
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog
required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region:
TestTable,0000073491,1270194953322
        at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:977)
        at
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:846)
        at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
        at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushSomeRegions(MemStoreFlusher.java:352)
        at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:321)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1775)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,384 WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush failed
2010-04-02 16:00:22,385 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,386 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,387 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,387 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,388 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,388 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,388 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,388 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,389 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,389 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020, call put([B@688c3d16,
[Lorg.apache.hadoop.hbase.client.Put;@4efea4f9) from 172.31.13.232:60424:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,389 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,390 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,390 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,390 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,390 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,391 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020, call put([B@42d1e0a3,
[Lorg.apache.hadoop.hbase.client.Put;@60f30f9a) from 172.31.13.232:60424:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,389 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 17 on 60020, call put([B@3000fff3,
[Lorg.apache.hadoop.hbase.client.Put;@11a3c324) from 172.31.13.215:36858:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,391 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 13 on 60020, call put([B@3a8edb93,
[Lorg.apache.hadoop.hbase.client.Put;@78f1aded) from 172.31.13.232:60424:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,392 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 10 on 60020, call put([B@2cdb1ba9,
[Lorg.apache.hadoop.hbase.client.Put;@64f50b40) from 172.31.13.232:60424:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,391 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020, call put([B@17cadb32,
[Lorg.apache.hadoop.hbase.client.Put;@6fcbbd2) from 172.31.13.215:36858:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,390 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,395 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,396 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 12 on 60020, call put([B@13b5ca88,
[Lorg.apache.hadoop.hbase.client.Put;@403b0d4) from 172.31.13.215:36858:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,396 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,397 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,397 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,397 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,398 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,398 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,399 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020, call put([B@2f48adb3,
[Lorg.apache.hadoop.hbase.client.Put;@4c83db05) from 172.31.13.232:60424:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,400 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,400 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,400 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 18 on 60020, call put([B@64a04c81,
[Lorg.apache.hadoop.hbase.client.Put;@57fedcc3) from 172.31.13.215:36858:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,401 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 22 on 60020, call put([B@1d4fb357,
[Lorg.apache.hadoop.hbase.client.Put;@5b8723c9) from 172.31.13.215:36858:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,402 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,402 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,403 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,403 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,403 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 16 on 60020, call put([B@4e644ec7,
[Lorg.apache.hadoop.hbase.client.Put;@1b701306) from 172.31.13.232:60424:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,404 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,404 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,405 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,405 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,406 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 21 on 60020, call put([B@5af0fcfc,
[Lorg.apache.hadoop.hbase.client.Put;@3713309) from 172.31.13.232:60424:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,406 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020, call put([B@7ccd95a2,
[Lorg.apache.hadoop.hbase.client.Put;@1d1afeb1) from 172.31.13.215:36858:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,407 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,407 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,408 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 14 on 60020, call put([B@57472564,
[Lorg.apache.hadoop.hbase.client.Put;@7fe7f4ff) from 172.31.13.232:60424:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,408 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 23 on 60020, call put([B@5585d2f3,
[Lorg.apache.hadoop.hbase.client.Put;@44dc7b62) from 172.31.13.215:36858:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,409 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 19 on 60020, call put([B@6728823f,
[Lorg.apache.hadoop.hbase.client.Put;@147f2b02) from 172.31.13.215:36858:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,409 FATAL org.apache.hadoop.hbase.regionserver.HLog:
Could not append. Requesting close of hlog
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,409 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,392 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020, call put([B@7585792f,
[Lorg.apache.hadoop.hbase.client.Put;@dd738f0) from 172.31.13.232:60424:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,410 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 20 on 60020, call put([B@31d4244e,
[Lorg.apache.hadoop.hbase.client.Put;@5e9b9670) from 172.31.13.215:36858:
error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad.
Aborting...
java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241)
2010-04-02 16:00:22,987 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2010-04-02 16:00:22,987 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2010-04-02 16:00:22,988 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2010-04-02 16:00:22,988 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020
2010-04-02 16:00:22,989 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2010-04-02 16:00:22,990 INFO org.apache.hadoop.hdfs.DFSClient: Could not
obtain block blk_-2614684857813896108_12085 from any node:
java.io.IOException: No live nodes contain current block
2010-04-02 16:00:22,987 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 10 on 60020: exiting
2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 11 on 60020: exiting
2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 12 on 60020: exiting
2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 13 on 60020: exiting
2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 14 on 60020: exiting
2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 16 on 60020: exiting
2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 17 on 60020: exiting
2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 18 on 60020: exiting
2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 19 on 60020: exiting
2010-04-02 16:00:22,993 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 21 on 60020: exiting
2010-04-02 16:00:22,997 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 22 on 60020: exiting
2010-04-02 16:00:22,997 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 23 on 60020: exiting
2010-04-02 16:00:22,998 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 24 on 60020: exiting
2010-04-02 16:00:22,998 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2010-04-02 16:00:23,000 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2010-04-02 16:00:23,001 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2010-04-02 16:00:23,001 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 20 on 60020: exiting



在 2010年4月2日 下午3:46,Chen Bangzhong <bangzhong@gmail.com>写道:

> Hi, All
>
> I am benchmarking hbase. My HDFS clusters includes 4 servers (Dell 860,
> with 2 GB RAM). One NameNode, one JobTracker, 2 DataNodes.
>
> My HBase Cluster also comprise 4 servers too. One Master, 2 region and one
> ZooKeeper. (Dell 860, with 2 GB RAM)
>
> I runned the org.apache.hadoop.PerformanceEvaluation on the ZooKeeper
> server. the ROW_LENGTH was changed from 1000 to ROW_LENGTH = 100*1024;
> So each value will be 100k in size.
>
> hadoop version is 0.20.2, hbase version is 0.20.3. dfs.replication set to
> 1.
>
> The following is the command line:
>
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=10000 randomWrite 20.
>
> It tooks about one hour to complete the test(3468628 ms), about 60 writes
> per second. It seems the performance is disappointing.
>
> Is there anything I can do to make hbase perform better under 100k size ?I
> didn't try the method mentioned in the performance wiki yet, because I
> thought 60writes/sec is too low.
>
> If the value size is 1k, hbase performs much better. 200000 sequencewrite
> tooks about 16 seconds, about 12500 writes/per second.
>
> Now I am trying to benchmark using two clients on 2 servers, no result yet.
>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message