hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Too many open files (java.net.SocketException)
Date Tue, 09 Apr 2013 15:33:12 GMT
Hi,

I just faced an issue this morning on one of my RS.

Here is an extract of the logs
2013-04-09 11:05:33,164 ERROR org.apache.hadoop.hdfs.DFSClient:
Exception closing file
/hbase/entry_proposed/ae4a5d72d4613728ddbcc5a64262371b/.tmp/ed6a0154ef714cd88faf26061cf248d3
: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
	at sun.nio.ch.Net.socket0(Native Method)
	at sun.nio.ch.Net.socket(Net.java:323)
	at sun.nio.ch.Net.socket(Net.java:316)
	at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:101)
	at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
	at java.nio.channels.SocketChannel.open(SocketChannel.java:142)
	at org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:58)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:3423)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3381)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829)

ulimit is unlimited on all my servers.

Is seems there was to many network connections opened. Is there
anything HBase can handle in such scenario? It's only hadoop in the
stacktrace, so I'm not sure.

Can this be related to nproc? I don't think so. I have another tool
running on the RS. Using low CPU, low bandwidth but making MANY
network HTTP connections...

Any suggestion?

JM

Mime
View raw message