mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Goldstein Lyor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SSHD-557) MINA SSHD deadlocks upon multiple concurrent HTTP downloads over locally-forwarded ports
Date Wed, 16 Sep 2015 04:40:46 GMT

    [ https://issues.apache.org/jira/browse/SSHD-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746851#comment-14746851
] 

Goldstein Lyor commented on SSHD-557:
-------------------------------------

I am not sure this is a good fix. It may remove the blocking wait, but then flush() is actually
doing busy-wait loop instead of asynchronous event wait. It may send quite a lot of zero-length
packets to the other side while it is waiting for space to become available. Also, I don't
see an explanation as to how the deadlock occurs if waitForSpace() is called from flush().
Also, while the busy-wait loop is run due to the zero length, there are unused, potentially
large buffers being created - see lines 131-138 in ChannelOutputStream:

{code:java}
if (total == length) {
    newBuffer(length);
} else {
    int leftover = total - length;
    newBuffer(Math.max(leftover, length));
    buffer.putRawBytes(buf.array(), pos - leftover, leftover);
    bufferLength = leftover;
}
{code}

> MINA SSHD deadlocks upon multiple concurrent HTTP downloads over locally-forwarded ports
> ----------------------------------------------------------------------------------------
>
>                 Key: SSHD-557
>                 URL: https://issues.apache.org/jira/browse/SSHD-557
>             Project: MINA SSHD
>          Issue Type: Bug
>    Affects Versions: 0.14.0, 1.0.0
>         Environment: Fedora 21, kernel 3.17.4-301, OpenJDK 1.8.0_51-b16
>            Reporter: Sam Gardner
>            Assignee: Goldstein Lyor
>              Labels: sshd
>             Fix For: 1.1.0
>
>         Attachments: AbstractUserAuth.java, setup-tunnels-remote.sh, setup-tunnels.sh,
ssh-load-remote.sh, ssh-load.sh, sshd-core-0.14.0-jar-with-dependencies.jar, sshd-core-1.0.0-jar-with-dependencies.jar
>
>
> SSHD-Core 1.0.0 deadlocks when multiple concurrent downloads are in progress over ChannelOutputStreams
> In both Mina-SSHD 0.14.0 and 1.0.0 the server will become deadlocked if multiple concurrent
downloads are started over a locally-forwarded TCP socket. In this state the process is still
alive and client connections stay open, but no data can be transmitted and new connections
will not be accepted - when inspected via telnet, the SSH port will not even return a banner.
> Inspection of the hung process using the {{jstack}} tool shows that each NIO worker thread
is in the following state: {noformat}
> "sshd-SshServer[36f6e879]-nio2-thread-4" #14 daemon prio=5 os_prio=0 tid=0x00007f219c009000
nid=0x34c6 in Object.wait() [0x00007f21c8439000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000000d799f9b0> (a org.apache.sshd.common.channel.Window)
>         at java.lang.Object.wait(Object.java:502)
>         at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:175)
>         - locked <0x00000000d799f9b0> (a org.apache.sshd.common.channel.Window)
>         at org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:126)
>         - locked <0x00000000d799fa68> (a org.apache.sshd.common.channel.ChannelOutputStream)
>         at org.apache.sshd.server.forward.TcpipServerChannel$1.messageReceived(TcpipServerChannel.java:156)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:220)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
>         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>         at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
>         at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>         at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
>         at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
>         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>         at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
>         at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>         at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
>         at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
>         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>         at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
>         at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>         at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
>         at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
>         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>         at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
>         at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>         at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
>         at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
>         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>         at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
>         at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>         at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
>         at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
>         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>         at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
>         at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>         at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
>         at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
>         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>         at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
>         at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>         at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
>         at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
>         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>         at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
>         at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>         at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
>         at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
>         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>         at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
>         at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>         at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>         at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
>         at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
>         at org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
>         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>         at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>         at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}
> These are the steps to reproduce:
> # Start up approximately 200 separate SSH sessions with one local forward to an HTTP
server:
> {{ssh -N -LXXXXX:<HTTP_SERVER>:80 account@<MINA_SSHD_SERVER>}}
> ## To simplify this process, I modified [org.apache.sshd.server.auth.AbstractUserAuth|^AbstractUserAuth.java]
to always accept requests - this makes it possible to create the sessions on the same client
(though I've also reproduced this on separate clients as well).
> ## This [script|^setup-tunnels.sh] will setup the client sessions as background processes
> # With the above sessions alive and backgrounded, fetch large files concurrently from
the HTTP server over multiple forwarded ports:
> {{wget http://localhost:XXXXX/hugefile}}
> ## This [script|^ssh-load.sh] will start {{wget}} downloads in the above manner.
> # After a time, the {{wget}} downloads will stop progressing, and inspection of the SSH
server process via {{jstack <PID>}} will show each of the {{nio2-thread-[0-9]}} threads
in the deadlocked state.
> Note: I previously filed SSHD-539 on this, and have since reproduced it on vanilla MINA
SSHD-Core without our proprietary modifications and running outside of Tomcat.
> The JAR files I used to reproduce are attached to this ticket as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message