qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (QPID-6649) Client initiated connection close can race with Broker shutdown (or VH mastership change) leading to exception and unclosed socket
Date Tue, 21 Jul 2015 13:38:05 GMT

    [ https://issues.apache.org/jira/browse/QPID-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635114#comment-14635114
] 

ASF subversion and git services commented on QPID-6649:
-------------------------------------------------------

Commit 1692114 from [~k-wall] in branch 'java/trunk'
[ https://svn.apache.org/r1692114 ]

QPID-6649: [Java Broker] Refactor connection close to avoid race conditions

* particularly deals with race condition occuring when a connection closed initiated by a
virtualhost (or broker)
  close races with a connection close initiated from the messaging client.

Work by Lorenz Quack <quack.lorenz@gmail.com> and Keith Wall

> Client initiated connection close can race with Broker shutdown (or VH mastership change)
leading to exception and unclosed socket
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: QPID-6649
>                 URL: https://issues.apache.org/jira/browse/QPID-6649
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>            Reporter: Keith Wall
>            Assignee: Keith Wall
>             Fix For: 6.0 [Java]
>
>         Attachments: 0001-QPID-6649-Ensure-socket-is-closed.-Make-QBTC-fail-if.patch,
0001-QPID-6649-Java-Broker-Refactor-connection-close-to-a.patch
>
>
> There is a potential for a race in the Java Broker that is being exposed by the System
Tests.  The unlucky case is as follows.  The client closes the connection by sending ConnectionClose.
On receiving this, ServerConnectionDelegate.connectionClose() invokes ServerConnection.setState()
which invokes AbstractVirtualHost.deregisterConnectionAsync().  This causes the AVH to 'forget'
the connection (removed from AVH#_connections) even though it is not yet closed.  Meanwhile,
QBTC begins to close the Broker.  The Broker closes the AVH which calls closeConnections -
but the connection is already for forgotten (_connections#size() == 0) so as far as it is
concerned there are no connections to await for, so the VH close concludes and the VHN shutdowns
the VH's executor.  Then the IO thread gets scheduled, sends the ConnectionCloseOk bytes down
the wire and then NBC#shutdown invokes MVPE#close.... which finally invokesServerConnection.setState
which calls AbstractVirtualHost.deregisterConnectionAsync, which fails (stack trace below)
because the task executor is already closed.
> This causes a socket to remain open.
> {noformat}
> java.lang.IllegalStateException: Task executor is not in ACTIVE state
>         at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.checkState(TaskExecutorImpl.java:310)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.submit(TaskExecutorImpl.java:141)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.model.AbstractConfiguredObject.doOnConfigThread(AbstractConfiguredObject.java:499)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.virtualhost.AbstractVirtualHost.deregisterConnectionAsync(AbstractVirtualHost.java:1753)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.virtualhost.AbstractVirtualHost.deregisterConnection(AbstractVirtualHost.java:1748)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.protocol.v0_10.ServerConnection.setState(ServerConnection.java:161)
~[qpid-broker-plugins-amqp-0-10-protocol-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.transport.Connection.closed(Connection.java:594) ~[qpid-common-6.0.0-SNAPSHOT.jar:na]
>         at org.apache.qpid.server.protocol.v0_10.ServerConnection.closed(ServerConnection.java:495)
~[qpid-broker-plugins-amqp-0-10-protocol-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.transport.network.Assembler.closed(Assembler.java:113) ~[qpid-common-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:215)
~[qpid-common-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.protocol.v0_10.AMQPConnection_0_10.closed(AMQPConnection_0_10.java:259)
~[qpid-broker-plugins-amqp-0-10-protocol-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.transport.MultiVersionProtocolEngine.closed(MultiVersionProtocolEngine.java:108)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.transport.NonBlockingConnection.shutdown(NonBlockingConnection.java:268)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:255)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:119)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.transport.NetworkConnectionScheduler.access$000(NetworkConnectionScheduler.java:37)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at org.apache.qpid.server.transport.NetworkConnectionScheduler$2.run(NetworkConnectionScheduler.java:100)
~[qpid-broker-core-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_65]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_65]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message