hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "聪聪" <175998...@qq.com>
Subject Query occasionally respond very slowly
Date Tue, 23 Jan 2018 06:50:33 GMT
Recently, query occasionally respond very slowly.These queries usually return quickly, within
a few milliseconds.But occasionally it gets very slow, reaching more than 20 seconds.I looked
at the gc log and there was no full gc happening.


A regionserver log is as follows:
2018-01-22 16:38:13,580 WARN  [B.defaultRpcServer.handler=35,queue=5,port=60020] ipc.RpcServer:
(responseTooSlow): {"processingtimems":23513,"call":"Get(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetRequest)","client":"10.94.76.216:34324","starttimems":1516610270064,"queuetimems":0,"class":"HRegionServer","responsesize":412,"method":"Get"}


One of the datanode logs is as follows:
2018-01-22 16:37:42,417 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.90.18.70:50010, dest: /10.90.18.70:54469, bytes: 12288, op: HDFS_READ, cliID: DFSClient_hb_rs_l-hbase50.dba.cn2.qunar.com,60020,1505725242560_-1708409423_37,
offset: 948224, srvID: ab75b2a1-af8b-4fcf-a93a-6245aab9241c, blockid: BP-1760821987-10.90.18.66-1447407547902:blk_1121353497_47612799,
duration: 9866301
2018-01-22 16:37:42,499 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1760821987-10.90.18.66-1447407547902:blk_1121355749_47615051
src: /10.90.18.69:36293 dest: /10.90.18.70:50010
2018-01-22 16:37:42,499 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
BP-1760821987-10.90.18.66-1447407547902:blk_1121355749_47615051 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException:
Block BP-1760821987-10.90.18.66-1447407547902:blk_1121355749_47615051 already exists in state
FINALIZED and thus cannot be created.
2018-01-22 16:37:42,499 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: l-hbase50.dba.cn2:50010:DataXceiver
error processing WRITE_BLOCK operation  src: /10.90.18.69:36293 dst: /10.90.18.70:50010; org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException:
Block BP-1760821987-10.90.18.66-1447407547902:blk_1121355749_47615051 already exists in state
FINALIZED and thus cannot be created.
2018-01-22 16:37:42,506 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.90.18.70:50010, dest: /10.90.18.70:54510, bytes: 12288, op: HDFS_READ, cliID: DFSClient_hb_rs_l-hbase50.dba.cn2.qunar.com,60020,1505725242560_-1708409423_37,
offset: 34276352, srvID: ab75b2a1-af8b-4fcf-a93a-6245aab9241c, blockid: BP-1760821987-10.90.18.66-1447407547902:blk_1121354564_47613866,
duration: 7418016







Another datanode log is as follows:
2018-01-22 16:37:42,497 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.90.18.69,
datanodeUuid=95aafbc6-239c-4661-ba37-4687ae9e663b, infoPort=50075, ipcPort=50020, storageInfo=lv=-56;cid=CID-1fa1156b-bd6f-4113-8d02-3af80df935c3;nsid=470632750;c=0)
Starting thread to transfer BP-1760821987-10.90.18.66-1447407547902:blk_1121355749_47615051
to 10.90.18.70:50010
2018-01-22 16:37:42,499 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.90.18.69,
datanodeUuid=95aafbc6-239c-4661-ba37-4687ae9e663b, infoPort=50075, ipcPort=50020, storageInfo=lv=-56;cid=CID-1fa1156b-bd6f-4113-8d02-3af80df935c3;nsid=470632750;c=0):Failed
to transfer BP-1760821987-10.90.18.66-1447407547902:blk_1121355749_47615051 to 10.90.18.70:50010
got
java.net.SocketException: Original Exception : java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:547)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:716)
        at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1805)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Connection reset by peer
        ... 8 more
2018-01-22 16:37:42,520 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.90.18.69:50010, dest: /10.90.18.69:49343, bytes: 14848, op: HDFS_READ, cliID: DFSClient_hb_rs_l-hbase49.dba.cn2.qunar.com,60020,1464835349894_1899722521_37,
offset: 61291520, srvID: 95aafbc6-239c-4661-ba37-4687ae9e663b, blockid: BP-1760821987-10.90.18.66-1447407547902:blk_1121217415_47476717,
duration: 5939553



This question confused me.What caused the problem?How do we solve this?
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message