hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Downing <tdown...@proteus-technologies.com>
Subject Re: High ingest rate and FIN_WAIT1 problems
Date Mon, 19 Jul 2010 09:13:48 GMT
Thanks for the response, but my problem is not with FIN_WAIT2, it
is with FIN_WAIT1.

If it was FIN_WAIT2, the only concern would be socket leakage,
and if  setting the time out solved the issue, that would be great.

The problem with FIN_WAIT1 is twofold - first, it is incumbent on
the application to notice and handle this problem; from the TCP stack
point of view, there is nothing wrong.  It is just a special case of slow
consumer.  The other problem is that it implies that something will be
lost if the socket is abandoned, there is data in the send queue of the
socket in FIN_WAIT1 that has not yet been delivered to the peer.

On 7/16/2010 3:56 PM, Ryan Rawson wrote:
> I've been running with this setting on both the HDFS side and the
> HBase side for over a year now, it's a bit of voodoo but you might be
> running into well known suckage of HDFS.  Try this one and restart
> your hbase&  hdfs.
>
> The FIN_WAIT2/TIME_WAIT happens more on large concurrent gets, not so
> much for inserts.
>
> <property>
> <name>dfs.datanode.socket.write.timeout</name>
> <value>0</value>
> </property>
>
> -ryan
>
>
> On Fri, Jul 16, 2010 at 9:33 AM, Thomas Downing
> <tdowning@proteus-technologies.com>  wrote:
>    
>> Thanks for the response.
>>
>> My understanding is that TCP_FIN_TIMEOUT affects only FIN_WAIT2,
>> my problem is with FIN_WAIT1.
>>
>> While I do see some sockets in TIME_WAIT, they are only a few, and the
>> number is not growing.
>>
>> On 7/16/2010 12:07 PM, Hegner, Travis wrote:
>>      
>>> Hi Thomas,
>>>
>>> I ran into a very similar issue when running slony-I on postgresql to
>>> replicate 15-20 databases.
>>>
>>> Adjusting the TCP_FIN_TIMEOUT parameters for the kernel may help to slow
>>> (or hopefully stop), the leaking sockets. I found some notes about adjusting
>>> TCP parameters here:
>>> http://www.hikaro.com/linux/tweaking-tcpip-syctl-conf.html
>>>
>>>        
[snip]

Mime
View raw message