mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Binay Paul (Jira)" <j...@apache.org>
Subject [jira] [Created] (SSHD-1025) "java.net.SocketException: Too many open files" exception on long run of application with Apache Mina SSHD 2.1.0
Date Sun, 28 Jun 2020 23:17:00 GMT
Binay Paul created SSHD-1025:
--------------------------------

             Summary: "java.net.SocketException: Too many open files" exception on long run
of application with Apache Mina SSHD 2.1.0
                 Key: SSHD-1025
                 URL: https://issues.apache.org/jira/browse/SSHD-1025
             Project: MINA SSHD
          Issue Type: Bug
    Affects Versions: 2.1.0
         Environment: Oracle Linux 7
            Reporter: Binay Paul


This issue is observed during the performance testing of Oracle Communications Sessions Delivery
Manager (OCSDM) release 8.2.1 with Apache Mina SSHD 2.1.0 version, after a long continuous
run for about 15 to 16 hours of OCSDM with processing REST/SOAP API calls in a 3 node cluster
setup.

Root Cause Analysis:

When OCSDMis running continuously with REST/SOAP consecutive requests, list of opened file
descriptors count increases rapidly like some where around 23 Lakh and this leads to an exception
"java.net.SocketException: Too many open files".

As a result of this DB read/write operations fails and OCSDM shutdown task is triggered by
the application.

The lsof command shows FD named "PIPE" and "EventPoll" are been opened by SDM process in large
number.

These FDs are opened by third party library Apache Mina which is used to make SSH/SFTP connections
to 297 devices which are present in SDM (Used for SDM performance testing).

Walk around  : As of now for OCSDM  release 8.2.1 library Apache Mina is replaced with JSCH
0.1.55 which is been used in previous versions of SDM and application works as expected.

 

Console.out :
 [Wed Nov 13 23:49:36 EST 2019] WARNING: RMI TCP Accept-1099: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,localport=1099]
throws
 [Wed Nov 13 23:49:36 EST 2019] java.net.SocketException: Too many open files
 [Wed Nov 13 23:49:36 EST 2019] at java.net.PlainSocketImpl.socketAccept(Native Method)
 [Wed Nov 13 23:49:36 EST 2019] at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
 [Wed Nov 13 23:49:36 EST 2019] at java.net.ServerSocket.implAccept(ServerSocket.java:545)
 [Wed Nov 13 23:49:36 EST 2019] at java.net.ServerSocket.accept(ServerSocket.java:513)
 [Wed Nov 13 23:49:36 EST 2019] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
 [Wed Nov 13 23:49:36 EST 2019] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
 [Wed Nov 13 23:49:36 EST 2019] at java.lang.Thread.run(Thread.java:745)

BEServer.log :

 

11/13 23:49:41.300 INFO [com.acmepacket.ems.server.main.BEServer] - Method: [shutdown] Thread:
[Thread-176494:334181] Msg:[##### BEServer shutdown was called ##### : Starting shutdown of
BE server. ]
 11/13 23:49:41.300 INFO [com.acmepacket.ems.server.main.BEServer] - Method: [shutDownApacheServer]
Thread: [Thread-176494:334181] Msg:[##### WebServer shutdown was called ##### : Starting shutdown
of http server]
 11/13 23:49:41.300 ERROR [com.acmepacket.ems.server.main.BEServer] - Method: [shutDownApacheServer]
Thread: [Thread-176494:334181] Msg:[Exception thrown in attempting to shutdown Web server.
Reason : Cannot run program "../Apache/httpse
 rver/bin/stopHttpServer.sh": error=24, Too many open files]
 java.io.IOException: Cannot run program "../Apache/httpserver/bin/stopHttpServer.sh": error=24,
Too many open files
 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_74]
 at java.lang.Runtime.exec(Runtime.java:620) ~[?:1.8.0_74]
 at java.lang.Runtime.exec(Runtime.java:450) ~[?:1.8.0_74]
 at java.lang.Runtime.exec(Runtime.java:347) ~[?:1.8.0_74]
 at com.acmepacket.ems.server.main.BEServer.shutDownApacheServer(BEServer.java:989) [ACMEPacketBEServer.jar:?]
 at com.acmepacket.ems.server.main.BEServer.shutdownWebServers(BEServer.java:954) [ACMEPacketBEServer.jar:?]
 at com.acmepacket.ems.server.main.BEServer.shutdown(BEServer.java:433) [ACMEPacketBEServer.jar:?]
 at com.acmepacket.ems.server.main.BEServer.shutdown(BEServer.java:403) [ACMEPacketBEServer.jar:?]
 at com.acmepacket.ems.server.services.database.ReplicatedXMLDatabaseManagerImpl.shutDownAfterFault(ReplicatedXMLDatabaseManagerImpl.java:2536)
[ACMEPacketBEServer.jar:?]
 at com.acmepacket.ems.server.services.database.ReplicatedXMLDatabaseManagerImpl.handlePanicEvent(ReplicatedXMLDatabaseManagerImpl.java:2568)
[ACMEPacketBEServer.jar:?]
 at com.sleepycat.db.internal.DbEnv.handle_panic_event_notify(DbEnv.java:115) [db.jar:?]
 Caused by: java.io.IOException: error=24, Too many open files
 at java.lang.UNIXProcess.forkAndExec(Native Method) ~[?:1.8.0_74]
 at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) ~[?:1.8.0_74]
 at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[?:1.8.0_74]
 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_74]
 ... 10 more



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org


Mime
View raw message