qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleksandr Rudyy <oru...@gmail.com>
Subject Re: Deadlock
Date Sat, 04 Aug 2012 01:44:11 GMT
Hi Eugene,

I do apologize for making you waiting with the answer. I was quite
busy with my current  tasks and did not have time to look through the
client  in order to recollect the deadlock conditions.

Regarding the deadlock scenario you have described, I believe that you
are right. I cannot see in Qpid code the conditions which might lead
into such deadlock by closing of  consumer inside of MessageListener
on 0.10 path. However, such deadlock  can  occur in a bit different
way and it is even reported in QPID-4086.

If your application is not using failover it seems it is safe.  At
least, I cannot see the deadlock conditions in the current code not
related to failover.

In case of failover,  potentially the dedlock can occur in the
following scenario when consumer is closed inside of MessageListener
(unless I missed something) :

1) Message is dispatched into MessageListener#onMessage(Message) but
MessageListener is trying to invoke the consumer close method. At this
moment in time, the dispatcher thread acquired the dispatcher lock
(AMQSession.Dispatcher._lock) and delivery lock
(AMQSession._deliveryLock) and is about to acquire the failover mutex
from BasicMessageConsumer#close()
2) At this time, the tcp connection is lost and failover is triggered
to restore the connectivity. On failover, the failover thread (which
is a receiver thread) acquires  the failover mutex before the
dispatcher thread. Than it restores the connectivity and invokes
AMQConnection#.failoverPrep()  to clean the data on connection (pls,
have a look into code in
AMQConnectionDelegate_0_10#closed(Connection))
3) AMQConnection#.failoverPrep() invokes 0.10 delegate which iterates
through the sessions and invokes failoverPrep() on each session, which
in turn calls syncDispatchQueue() method to clean the dispatcher
queue. syncDispatchQueue() sends the poisoned message into the
pre-fetched message queue and waits forever for message being consumed
in the dispatcher thread.
4) However, the dispatcher thread cannot process any message as it is
blocked in BasicMessageConsumer#close(). Thus, we have a dead lock.


Kind Regards,
Alex

On 2 August 2012 12:11, eugene <eugen.rabii@gmail.com> wrote:
> Well this is a bug in the libraries, I've been looking at the code and here
> is (probably obvious for you :) ) problem:
>
> 1. Thread 1 acquires the FailoverMutex
> 2. Thread 2 acquires the MessageDeliveryLock
> 3. Thread 1 tries to acquire the MessageDeliveryLock
> 4. Thread 2 tries to acquire the FailoverMutex
>
> Thus the deadlock.
>
> Alex, I do not think that it has anything to do with closing Consumers from
> Listeners, it's just a side effect.
>
> Thank You,
> Eugene.
>
>
>
> --
> View this message in context: http://apache-qpid-developers.2158895.n2.nabble.com/Deadlock-tp7581518p7581564.html
> Sent from the Apache Qpid developers mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> For additional commands, e-mail: dev-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message