mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lyor Goldstein (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SSHD-1025) "java.net.SocketException: Too many open files" exception on long run of application with Apache Mina SSHD 2.1.0
Date Mon, 29 Jun 2020 16:16:00 GMT

    [ https://issues.apache.org/jira/browse/SSHD-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147915#comment-17147915
] 

Lyor Goldstein commented on SSHD-1025:
--------------------------------------

It's hard to tell whether this is an SSHD issue or a code mis-use - i.e., is it the code using
SSHD that does not release sessions or is it SSHD itself that fails to release them. In any
case, version 2.1.0 is pretty old - we recently release 2.5.0 and are probably releasing 2.5.1
to fix an SFTP issue. I recommend you try 2.5.0 (or at the very least 2.4.0) and see if this
issue is re-produced.

> "java.net.SocketException: Too many open files" exception on long run of application
with Apache Mina SSHD 2.1.0
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: SSHD-1025
>                 URL: https://issues.apache.org/jira/browse/SSHD-1025
>             Project: MINA SSHD
>          Issue Type: Bug
>    Affects Versions: 2.1.0
>         Environment: Oracle Linux 7
>            Reporter: Binay Paul
>            Priority: Major
>
> This issue is observed during the performance testing of Oracle Communications Sessions
Delivery Manager (OCSDM) release 8.2.1 with Apache Mina SSHD 2.1.0 version, after a long continuous
run for about 15 to 16 hours of OCSDM with processing REST/SOAP API calls in a 3 node cluster
setup.
> Root Cause Analysis:
> When OCSDMis running continuously with REST/SOAP consecutive requests, list of opened
file descriptors count increases rapidly like some where around 23 Lakh and this leads to
an exception "java.net.SocketException: Too many open files".
> As a result of this DB read/write operations fails and OCSDM shutdown task is triggered
by the application.
> The lsof command shows FD named "PIPE" and "EventPoll" are been opened by SDM process
in large number.
> These FDs are opened by third party library Apache Mina which is used to make SSH/SFTP
connections to 297 devices which are present in SDM (Used for SDM performance testing).
> Walk around  : As of now for OCSDM  release 8.2.1 library Apache Mina is replaced with
JSCH 0.1.55 which is been used in previous versions of SDM and application works as expected.
>  
> Console.out :
>  [Wed Nov 13 23:49:36 EST 2019] WARNING: RMI TCP Accept-1099: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,localport=1099]
throws
>  [Wed Nov 13 23:49:36 EST 2019] java.net.SocketException: Too many open files
>  [Wed Nov 13 23:49:36 EST 2019] at java.net.PlainSocketImpl.socketAccept(Native Method)
>  [Wed Nov 13 23:49:36 EST 2019] at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
>  [Wed Nov 13 23:49:36 EST 2019] at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>  [Wed Nov 13 23:49:36 EST 2019] at java.net.ServerSocket.accept(ServerSocket.java:513)
>  [Wed Nov 13 23:49:36 EST 2019] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
>  [Wed Nov 13 23:49:36 EST 2019] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
>  [Wed Nov 13 23:49:36 EST 2019] at java.lang.Thread.run(Thread.java:745)
> BEServer.log :
>  
> 11/13 23:49:41.300 INFO [com.acmepacket.ems.server.main.BEServer] - Method: [shutdown]
Thread: [Thread-176494:334181] Msg:[##### BEServer shutdown was called ##### : Starting shutdown
of BE server. ]
>  11/13 23:49:41.300 INFO [com.acmepacket.ems.server.main.BEServer] - Method: [shutDownApacheServer]
Thread: [Thread-176494:334181] Msg:[##### WebServer shutdown was called ##### : Starting shutdown
of http server]
>  11/13 23:49:41.300 ERROR [com.acmepacket.ems.server.main.BEServer] - Method: [shutDownApacheServer]
Thread: [Thread-176494:334181] Msg:[Exception thrown in attempting to shutdown Web server.
Reason : Cannot run program "../Apache/httpse
>  rver/bin/stopHttpServer.sh": error=24, Too many open files]
>  java.io.IOException: Cannot run program "../Apache/httpserver/bin/stopHttpServer.sh":
error=24, Too many open files
>  at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_74]
>  at java.lang.Runtime.exec(Runtime.java:620) ~[?:1.8.0_74]
>  at java.lang.Runtime.exec(Runtime.java:450) ~[?:1.8.0_74]
>  at java.lang.Runtime.exec(Runtime.java:347) ~[?:1.8.0_74]
>  at com.acmepacket.ems.server.main.BEServer.shutDownApacheServer(BEServer.java:989) [ACMEPacketBEServer.jar:?]
>  at com.acmepacket.ems.server.main.BEServer.shutdownWebServers(BEServer.java:954) [ACMEPacketBEServer.jar:?]
>  at com.acmepacket.ems.server.main.BEServer.shutdown(BEServer.java:433) [ACMEPacketBEServer.jar:?]
>  at com.acmepacket.ems.server.main.BEServer.shutdown(BEServer.java:403) [ACMEPacketBEServer.jar:?]
>  at com.acmepacket.ems.server.services.database.ReplicatedXMLDatabaseManagerImpl.shutDownAfterFault(ReplicatedXMLDatabaseManagerImpl.java:2536)
[ACMEPacketBEServer.jar:?]
>  at com.acmepacket.ems.server.services.database.ReplicatedXMLDatabaseManagerImpl.handlePanicEvent(ReplicatedXMLDatabaseManagerImpl.java:2568)
[ACMEPacketBEServer.jar:?]
>  at com.sleepycat.db.internal.DbEnv.handle_panic_event_notify(DbEnv.java:115) [db.jar:?]
>  Caused by: java.io.IOException: error=24, Too many open files
>  at java.lang.UNIXProcess.forkAndExec(Native Method) ~[?:1.8.0_74]
>  at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) ~[?:1.8.0_74]
>  at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[?:1.8.0_74]
>  at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_74]
>  ... 10 more



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org


Mime
View raw message