qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rob Godfrey (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (QPID-6813) Consumer tasks may run on the Broker thread leading to deadlock
Date Mon, 02 Nov 2015 17:19:27 GMT

     [ https://issues.apache.org/jira/browse/QPID-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rob Godfrey resolved QPID-6813.
-------------------------------
    Resolution: Fixed

> Consumer tasks may run on the Broker thread leading to deadlock
> ---------------------------------------------------------------
>
>                 Key: QPID-6813
>                 URL: https://issues.apache.org/jira/browse/QPID-6813
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>    Affects Versions: qpid-java-6.0
>            Reporter: Keith Wall
>            Assignee: Rob Godfrey
>            Priority: Blocker
>             Fix For: qpid-java-6.0
>
>
> Since QPID-6597, a consumer's tasks may run on the Broker's task executor rather than
the Virtualhost's as was intended.  This may lead to deadlock.
> The problem occurs because ACO discovers its task executor from its parents.  If a ACO
has multiple parents, the chosen parent is non-deterministic.  This didn't used to matter
because Consumers parents Queue and Session both shared a Virtualhost ancestor, so the Virtualhost
task executor was always selected.  However, QPID-6597, re-homed Connection (the parent of
Session) to be a child of Broker.  Since that point the executor used has been non-deterministic,
Consumers may end up with a Broker or Virtualhost task executor.
> Testing with very large numbers of connections exposed a deadlock between the Broker
and Virtualhost configuration thread.
> {code:java}
> protected AbstractConfiguredObject(final Map<Class<? extends ConfiguredObject>,
ConfiguredObject<?>> parents,
>                                        Map<String, Object> attributes)
>     {
>         this(parents, attributes, parents.values().iterator().next().getChildExecutor());
>     }
> {code}
> (The VH thread running an addConsumer blocks opening a Consumer using the Broker thread,
meanwhile the Broker thread *running a consumer close* tries to use the VH executor to remove
a queue. Deadlock.) 
> {noformat}
> "VirtualHostNode-default-Config" prio=10 tid=0x00007f48701a0000 nid=0xd714 waiting on
condition [0x00007f48ac728000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000e7ecd4a8> (a com.google.common.util.concurrent.AbstractFuture$Sync)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>         at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
>         at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
>         at org.apache.qpid.server.model.AbstractConfiguredObject.doSync(AbstractConfiguredObject.java:1937)
>         at org.apache.qpid.server.model.AbstractConfiguredObject.open(AbstractConfiguredObject.java:452)
>         at org.apache.qpid.server.queue.QueueConsumerImpl.<init>(QueueConsumerImpl.java:136)
>         at org.apache.qpid.server.queue.AbstractQueue.addConsumerInternal(AbstractQueue.java:864)
>         at org.apache.qpid.server.queue.AbstractQueue.access$200(AbstractQueue.java:119)
>         at org.apache.qpid.server.queue.AbstractQueue$6.execute(AbstractQueue.java:695)
>         at org.apache.qpid.server.queue.AbstractQueue$6.execute(AbstractQueue.java:691)
>         at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$TaskLoggingWrapper.execute(TaskExecutorImpl.java:270)
>         at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper$1.run(TaskExecutorImpl.java:342)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:356)
>         at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper.call(TaskExecutorImpl.java:335)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>  
>    Locked ownable synchronizers:
>         - <0x00000000e5953fb0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
>  
> "Broker-Config" prio=10 tid=0x00007f49c8f67000 nid=0xd70d waiting on condition [0x00007f48ac95e000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000ea9a2940> (a com.google.common.util.concurrent.AbstractFuture$Sync)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>         at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
>         at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
>         at org.apache.qpid.server.model.AbstractConfiguredObject.doSync(AbstractConfiguredObject.java:1937)
>         at org.apache.qpid.server.virtualhost.AbstractVirtualHost.removeQueue(AbstractVirtualHost.java:734)
>         at org.apache.qpid.server.queue.AbstractQueue$7.run(AbstractQueue.java:971)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:356)
>         at org.apache.qpid.server.queue.AbstractQueue.unregisterConsumer(AbstractQueue.java:966)
>         at org.apache.qpid.server.queue.QueueConsumerImpl.onClose(QueueConsumerImpl.java:282)
>         at org.apache.qpid.server.model.AbstractConfiguredObject$5$1.run(AbstractConfiguredObject.java:666)
>         at org.apache.qpid.server.model.AbstractConfiguredObject$19.onSuccess(AbstractConfiguredObject.java:2145)
>         at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319)
>         at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:422)
>         at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:172)
>         at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
>         at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)
>         at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)
>         at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1322)
>         at org.apache.qpid.server.model.AbstractConfiguredObject.doAfter(AbstractConfiguredObject.java:2138)
>         at org.apache.qpid.server.model.AbstractConfiguredObject$ChainedSettableFuture.then(AbstractConfiguredObject.java:2200)
>         at org.apache.qpid.server.model.AbstractConfiguredObject$5.execute(AbstractConfiguredObject.java:654)
>         at org.apache.qpid.server.model.AbstractConfiguredObject$5.execute(AbstractConfiguredObject.java:645)
>         at org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:520)
>         at org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:513)
>         at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$TaskLoggingWrapper.execute(TaskExecutorImpl.java:270)
>         at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper$1.run(TaskExecutorImpl.java:342)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:356)
>         at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper.call(TaskExecutorImpl.java:335)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>  
>    Locked ownable synchronizers:
>         - <0x00000000e59603c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
>         - <0x00000000ea413e18> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message