mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hugo Ares <hugo.a...@ericsson.com>
Subject Re: [jira] [Commented] (SSHD-779) Deadlock in ChannelOutputStream
Date Tue, 12 Dec 2017 23:32:24 GMT
I never investigated the issues deeply enough so I could write a proper
bug report.We have been using the Nio2ServiceFactoryFactory for the
last year and a half so I do not recal all the issues.

Since we switched back to MinaServiceFactoryFactory because of the
deadlock, the only issue we keep having few times per day is the ssh
connection is accepted from the client perspective but the server never
process it so it hang until the timeout is reached at the reverse proxy
and client get disconnected.

I did not create a bug report yet because I need to gather more
information otherwise it will be impossible to fix.

On Fri, 2017-12-08 at 19:02 +0000, Jonathan Valliere wrote:
> You said “less major issues”; are there bugs reported for
> MinaServiceFactoryFactory?
> 
> On Fri, Dec 8, 2017 at 1:22 PM Hugo Arès (JIRA) <jira@apache.org>
> wrote:
> 
> > 
> > 
> >     [
> > https://issues.apache.org/jira/browse/SSHD-779?page=com.atlassian.j
> > ira.plugin.system.issuetabpanels:comment-
> > tabpanel&focusedCommentId=16283975#comment-16283975
> > ]
> > 
> > Hugo Arès commented on SSHD-779:
> > --------------------------------
> > 
> > By nio2, I mean if we use Nio2ServiceFactoryFactory as the
> > IoServiceFactoryFactory.
> > 
> > This deadlock is not happening with MinaServiceFactoryFactory as
> > the
> > IoServiceFactoryFactory.
> > 
> > We have other less major issues with MinaServiceFactoryFactory so
> > we
> > currently use this one in production but we will go back to
> > Nio2ServiceFactoryFactory when the deadlock is fixed.
> > 
> > > 
> > > Deadlock in ChannelOutputStream
> > > -------------------------------
> > > 
> > >                 Key: SSHD-779
> > >                 URL: https://issues.apache.org/jira/browse/SSHD-7
> > > 79
> > >             Project: MINA SSHD
> > >          Issue Type: Bug
> > >    Affects Versions: 1.4.0, 1.6.0
> > >            Reporter: Hugo Arès
> > > 
> > > Since Gerrit upgraded to 1.4.0, the following deadlock is
> > > happening with
> > NI02:
> > > 
> > > {noformat}
> > > Found one Java-level deadlock:
> > > =============================
> > > "SSH-Stream-Worker-20":
> > >   waiting to lock monitor 0x00007fac34021cf8 (object
> > > 0x00007fb450addb38,
> > a java.lang.Object),
> > > 
> > >   which is held by "sshd-SshServer[4c110f87]-nio2-thread-20"
> > > "sshd-SshServer[4c110f87]-nio2-thread-20":
> > >   waiting to lock monitor 0x00007fad20048678 (object
> > > 0x00007fb450ade608,
> > a org.apache.sshd.common.channel.ChannelOutputStream),
> > > 
> > >   which is held by "SSH-Stream-Worker-20"
> > > Java stack information for the threads listed above:
> > > ===================================================
> > > "SSH-Stream-Worker-20":
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSession.doWritePacke
> > t(AbstractSession.java:1056)
> > > 
> > >         - waiting to lock <0x00007fb450addb38> (a
> > > java.lang.Object)
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSession.writePacket(
> > AbstractSession.java:1005)
> > > 
> > >         at
> > org.apache.sshd.common.channel.AbstractChannel.writePacket(Abstract
> > Channel.java:766)
> > > 
> > >         at
> > org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOut
> > putStream.java:219)
> > > 
> > >         - locked <0x00007fb450ade608> (a
> > org.apache.sshd.common.channel.ChannelOutputStream)
> > > 
> > >         at
> > > sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
> > >         at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
> > >         - locked <0x00007fb450ae2eb0> (a
> > > java.io.OutputStreamWriter)
> > >         at
> > > java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
> > >         at java.io.BufferedWriter.flush(BufferedWriter.java:254)
> > >         - locked <0x00007fb450ae2eb0> (a
> > > java.io.OutputStreamWriter)
> > >         at java.io.PrintWriter.flush(PrintWriter.java:320)
> > >         - locked <0x00007fb450ae2e70> (a java.io.BufferedWriter)
> > >         at java.io.PrintWriter.checkError(PrintWriter.java:357)
> > >         at
> > com.google.gerrit.sshd.commands.StreamEvents.writeEvents(StreamEven
> > ts.java:234)
> > > 
> > >         at
> > com.google.gerrit.sshd.commands.StreamEvents.access$000(StreamEvent
> > s.java:53)
> > > 
> > >         at
> > com.google.gerrit.sshd.commands.StreamEvents$1.run(StreamEvents.jav
> > a:100)
> > > 
> > >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
> > 511)
> > > 
> > >         at
> > > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas
> > k.access$201(ScheduledThreadPoolExecutor.java:180)
> > > 
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas
> > k.run(ScheduledThreadPoolExecutor.java:293)
> > > 
> > >         at
> > com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:418)
> > > 
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
> > r.java:1142)
> > > 
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
> > or.java:617)
> > > 
> > >         at java.lang.Thread.run(Thread.java:748)
> > > "sshd-SshServer[4c110f87]-nio2-thread-20":
> > >         at
> > org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOut
> > putStream.java:241)
> > > 
> > >         - waiting to lock <0x00007fb450ade608> (a
> > org.apache.sshd.common.channel.ChannelOutputStream)
> > > 
> > >         at
> > org.apache.sshd.common.util.io.IoUtils.closeQuietly(IoUtils.java:13
> > 7)
> > > 
> > >         at
> > org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(Ch
> > annelSession.java:201)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.AbstractCloseable.close(Abstr
> > actCloseable.java:81)
> > > 
> > >         at
> > org.apache.sshd.common.channel.AbstractChannel.close(AbstractChanne
> > l.java:548)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(Par
> > allelCloseable.java:61)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleC
> > loseable.java:63)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doClos
> > eImmediately(AbstractInnerCloseable.java:45)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.AbstractCloseable.close(Abstr
> > actCloseable.java:81)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(Par
> > allelCloseable.java:61)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleC
> > loseable.java:63)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.SequentialCloseable$1.operati
> > onComplete(SequentialCloseable.java:55)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.SequentialCloseable$1.operati
> > onComplete(SequentialCloseable.java:45)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.SequentialCloseable.doClose(S
> > equentialCloseable.java:68)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleC
> > loseable.java:63)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doClos
> > eImmediately(AbstractInnerCloseable.java:45)
> > > 
> > >         at
> > org.apache.sshd.common.util.closeable.AbstractCloseable.close(Abstr
> > actCloseable.java:81)
> > > 
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSession.exceptionCau
> > ght(AbstractSession.java:862)
> > > 
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.exc
> > eptionCaught(AbstractSessionIoHandler.java:56)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2Session.exceptionCaught(Nio2Session.java:164)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2Session.handleWriteCycleFailure(Nio2Session.java:386)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2Session$2.onFailed(Nio2Session.java:348)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2CompletionHandler.lambda$failed$1(Nio2CompletionHandler.j
> > ava:46)
> > > 
> > >         at
> > > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$318/
> > > 1992024659.run(Unknown
> > Source)
> > > 
> > >         at java.security.AccessController.doPrivileged(Native
> > > Method)
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:45)
> > > 
> > >         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128)
> > >         at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> > >         at sun.nio.ch
> > .UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocket
> > ChannelImpl.java:736)
> > > 
> > >         at sun.nio.ch
> > .AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.
> > java:382)
> > > 
> > >         at sun.nio.ch
> > .AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.
> > java:399)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2Session.doWriteCycle(Nio2Session.java:334)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2Session.startWriting(Nio2Session.java:318)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2Session.write(Nio2Session.java:148)
> > > 
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSession.doWritePacke
> > t(AbstractSession.java:1063)
> > > 
> > >         - locked <0x00007fb450addb38> (a java.lang.Object)
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSession.writePacket(
> > AbstractSession.java:1005)
> > > 
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractConnectionService.se
> > ndGlobalResponse(AbstractConnectionService.java:694)
> > > 
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractConnectionService.gl
> > obalRequest(AbstractConnectionService.java:662)
> > > 
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractConnectionService.pr
> > ocess(AbstractConnectionService.java:353)
> > > 
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSession.doHandleMess
> > age(AbstractSession.java:564)
> > > 
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSession.handleMessag
> > e(AbstractSession.java:497)
> > > 
> > >         - locked <0x00007fb450ad9c48> (a java.lang.Object)
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSession.decode(Abstr
> > actSession.java:1406)
> > > 
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSession.messageRecei
> > ved(AbstractSession.java:458)
> > > 
> > >         - locked <0x00007fb450addb48> (a java.lang.Object)
> > >         at
> > org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.mes
> > sageReceived(AbstractSessionIoHandler.java:67)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:277)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2Session$1.onCompleted(Nio2Session.java:257)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2Session$1.onCompleted(Nio2Session.java:254)
> > > 
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandle
> > r.java:38)
> > > 
> > >         at
> > > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$182/
> > > 2116066305.run(Unknown
> > Source)
> > > 
> > >         at java.security.AccessController.doPrivileged(Native
> > > Method)
> > >         at org.apache.sshd.common.io
> > .nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37
> > )
> > > 
> > >         at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> > >         at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> > >         at sun.nio.ch
> > .AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.ja
> > va:112)
> > > 
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
> > r.java:1142)
> > > 
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
> > or.java:617)
> > > 
> > >         at java.lang.Thread.run(Thread.java:748)
> > > {noformat}
> > 
> > 
> > --
> > This message was sent by Atlassian JIRA
> > (v6.4.14#64029)
> > 
Mime
View raw message