hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 徐景辉 <xu...@neusoft.com>
Subject Hbase does not closing a closed socket resulting in thousand of CLOSE_WAIT sockets
Date Thu, 28 Aug 2014 09:18:28 GMT

I got the fatal error: HBase dose not close a dead connection with the
datanode. And I have reviewed the codes of Hadoop 2.4.0/HBase 0.98.0. I
found the patched codes of
<https://issues.apache.org/jira/browse/HBASE-9393> HBASE-9393 and
<https://issues.apache.org/jira/browse/HDFS-5671> HDFS-5671 have added into
the release of Hadoop 2.4.0/HBase 0.98.0. These codes are in the end of
method getRemoteBlockReaderFromTcp() in class BlockReaderFactory:

      } finally {

        if (blockReader == null) {

          IOUtils.cleanup(LOG, peer);



So I guess a new bug lead my problem and I create a new issue
<https://issues.apache.org/jira/browse/HBASE-11833> HBASE-11833. Do you guys
got the same issue? If that, please let me know. Thanks a lot.


Problem Description:
This resulting in over 30K+ CLOSE_WAIT and at some point HBase can not
connect to the datanode because too many mapped sockets from one host to
another on the same port:50010. 
After I restart all RSs, the count of CLOSE_WAIT will increase always.
$ netstat -an|grep CLOSE_WAIT|wc -l

1.    netstat -nap|grep CLOSE_WAIT|grep 6569|wc -l

2.    ps -ef|grep 6569
hbase 6569 6556 21 Aug25 ? 09:52:33 /opt/jdk1.6.0_25/bin/java
-Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m

I aslo have reviewed these issues:
 <https://issues.apache.org/jira/browse/HBASE-9393> HBASE-9393
 <https://issues.apache.org/jira/browse/HDFS-5671> HDFS-5671
 <https://issues.apache.org/jira/browse/HDFS-1836> HDFS-1836



Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)

is intended only for the use of the intended recipient and may be confidential and/or privileged
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication
not the intended recipient, unauthorized use, forwarding, printing,  storing, disclosure or
is strictly prohibited, and may be unlawful.If you have received this communication in error,please

immediately notify the sender by return e-mail, and delete the original message and all copies
your system. Thank you. 
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message