mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bill Kuker (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SSHD-721) deadlock: all nio workers wait to be woken up
Date Fri, 13 Apr 2018 12:37:00 GMT

    [ https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437253#comment-16437253
] 

Bill Kuker edited comment on SSHD-721 at 4/13/18 12:36 PM:
-----------------------------------------------------------

You know, as I am waking up this morning, I do remember noticing that in the change from 0.3.0
to 1.2.0 the port forwards and the client-server communication moved onto the same set of
workers.

{{web browser < - > ssh server < - > ssh client < - > web server}}

Could it be that simple: All nio workers are working on communication between the web browser
and ssh server, but can not make progress without communication between the ssh server and
ssh client, but no nio worker is available to move that data?


was (Author: bkuker1):
You know, as I am waking up this morning, I do remember noticing that in the change from 0.3.0
to 1.2.0 the port forwards and the client-server communication moved onto the same thread
pool.

{{web browser < - > ssh server < - > ssh client < - > web server}}

Could it be that simple: All nio workers are working on communication between the web browser
and ssh server, but can not make progress without communication between the ssh server and
ssh client, but no nio worker is available to move that data?

> deadlock: all nio workers wait to be woken up
> ---------------------------------------------
>
>                 Key: SSHD-721
>                 URL: https://issues.apache.org/jira/browse/SSHD-721
>             Project: MINA SSHD
>          Issue Type: Bug
>    Affects Versions: 1.3.0, 1.4.0
>            Reporter: Markus Rathgeb
>            Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming connections and port
forwarding requests.
> There are client machines (C) that run servers that should be accessible by a tunnel
to the server.
> On the client machines (C) also an implementation using sshd-core is running that establish
the connection to the server (S) and initiate the port forwarding.
> Other clients are using the tunnelled connection to communicate with the servers that
are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>    java.lang.Thread.State: TIMED_WAITING
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <132c6b60> (a java.lang.Object)
> 	at org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
> 	at org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
> 	at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
> 	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
> 	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
Source)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> 	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> 	at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> 	at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
> 	- locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>    java.lang.Thread.State: TIMED_WAITING
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <7e9f4eff> (a java.lang.Object)
> 	at org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
> 	at org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
> 	at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
> 	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
> 	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
Source)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> 	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> 	at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> 	at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
> 	- locked <35fbf3e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-1" - Thread t@122
>    java.lang.Thread.State: TIMED_WAITING
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <49ce93a9> (a java.lang.Object)
> 	at org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
> 	at org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
> 	at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
> 	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
> 	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
Source)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> 	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> 	at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> 	at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> 	at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> 	at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> 	at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:304)
> 	at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:249)
> 	at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:243)
> 	at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:239)
> 	at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:235)
> 	at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:231)
> 	at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:227)
> 	at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:178)
> 	at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:156)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
Source)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> 	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> 	at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> 	at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
> 	- locked <7d1c59e6> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-timer-thread-1" - Thread t@105
>    java.lang.Thread.State: TIMED_WAITING
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for <655c080c> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
> - None
> {noformat}
> To eliminate some other code that could trigger that error I created a "minimal" example
-- a simple test application -- that could be used to demonstrate the hang (for me it is reproducible
using that code).
> Please have a look at https://github.com/maggu2810/sshd-deadlock/tree/first-report where
you could also find a readme with a short description about the code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message