qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rupert Smith" <rupertlssm...@googlemail.com>
Subject Re: [java] Deadlock issue in trunk
Date Mon, 17 Sep 2007 14:38:05 GMT
Sorry, I'm talking rubbish. The ordering of the locks in AMQSession.close is
fine, and as it should be. Message delivery, followed by failover mutex.

The problematic interaction is because the call to AMQConnection.close,
obtains the failover mutex first. I'd be tempted to try and eliminate that
one, so that as each session is closed delivery then failover is
obtained/released for each.

You could remove the failover mutex from createTextMessage, but other
methods that do need to obtain it can be called from within a dispathcher
thread, so the problem can still occur.

Rupert

On 17/09/2007, Rupert Smith <rupertlssmith@googlemail.com> wrote:
>
> Also messageDeliveryLock seems like a poor choice of name.
> 'sessionCloseMutex' might be better.
>
> On 17/09/2007, Rupert Smith < rupertlssmith@googlemail.com> wrote:
> >
> > Actually, I see it now, the message delivery lock is obtained in the
> > dispatcher run method.
> >
> > So createTextMessage is doing:
> >
> > 1. Obtain message delivery lock.
> > 2. Obtain failover mutex.
> >
> > And the main thread calling close is doing:
> >
> > 1. Obtain failover mutex.
> > 2. Obtain message delivery lock.
> >
> > Just swap the lock ordering as I suggested. Unless this conflicts with
> > other orderings...
> >
> > On 17/09/2007, Rupert Smith <rupertlssmith@googlemail.com> wrote:
> > >
> > > It looks to me that the message delivery lock is the more fundamental
> > > cause of the dead lock. I can't figure out from the stack trace why the
> > > dispatcher thread is holding that one (I suppose I should look at what the
> > > test is doing to figure that out).
> > >
> > > Can we swap the order of obtaining the locks in AMQSession.close(long
> > > timeout) from
> > >
> > > synchronized(_messageDeliveryLock)
> > > synchronized (_connection.getFailoverMutex())
> > >
> > > to
> > >
> > > synchronized (_connection.getFailoverMutex())
> > > synchronized(_messageDeliveryLock)
> > >
> > > ?
> > >
> > > Unless that causes a different dead lock, it looks like it will fix
> > > the problem.
> > >
> > > Also, it looks to me that removing the failover mutex acquisition from
> > > createTextMessage will be harmless enough.
> > >
> > > Rupert
> > >
> > >
> > > On 15/09/2007, Rajith Attapattu < rajith77@gmail.com> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > When I run the latest version of trunk I get a hang on
> > > > org.apache.qpid.test.unit.client.forwardall.CombinedTest
> > > > kill -3 (thread dump) provides information to say there is a
> > > > deadlock.
> > > >
> > > > I also noted that all AMQSession.createXXXMessage contains
> > > > synchronization
> > > > code.
> > > > (in this case the deadlock happens inn AMQSession.createTextMessagetrying
> > > > to obtain the failover mutex)
> > > >
> > > > Do we really need this synchronization code when creating messages?
> > > > I am not
> > > > really convinced it is needed.
> > > > Can someody explain the reason for that code block?
> > > >
> > > > Regard,
> > > >
> > > > Rajith
> > > >
> > > > I have attached the trace below.
> > > >
> > > > Found one Java-level deadlock:
> > > > =============================
> > > > "Dispatcher-Channel-1":
> > > >   waiting to lock monitor 0x09ff950c (object 0x8b85a1a0, a
> > > > java.lang.Object
> > > > ),
> > > >   which is held by "main"
> > > > "main":
> > > >   waiting to lock monitor 0x09ff964c (object 0x88ff0240, a
> > > > java.lang.Object
> > > > ),
> > > >   which is held by "Dispatcher-Channel-1"
> > > >
> > > > Java stack information for the threads listed above:
> > > > ===================================================
> > > > "Dispatcher-Channel-1":
> > > >         at org.apache.qpid.client.AMQSession.createTextMessage(
> > > > AMQSession.java:1057)
> > > >         - waiting to lock <0x8b85a1a0> (a java.lang.Object)
> > > >         at org.apache.qpid.client.AMQSession.createTextMessage (
> > > > AMQSession.java:1068)
> > > >         at
> > > > org.apache.qpid.test.unit.client.forwardall.Service.onMessage(
> > > > Service.java:59)
> > > >         at org.apache.qpid.client.BasicMessageConsumer.notifyMessage
> > > > (
> > > > BasicMessageConsumer.java :628)
> > > >         at org.apache.qpid.client.BasicMessageConsumer.notifyMessage
> > > > (
> > > > BasicMessageConsumer.java:583)
> > > >         at
> > > > org.apache.qpid.client.AMQSession$Dispatcher.dispatchMessage(
> > > > AMQSession.java:2579)
> > > >         at org.apache.qpid.client.AMQSession$Dispatcher.run(
> > > > AMQSession.java
> > > > :2502)
> > > >         - locked <0x88ff0240> (a java.lang.Object)
> > > >         - locked <0x88ff0238> (a java.lang.Object)
> > > > "main":
> > > >         at org.apache.qpid.client.AMQSession.close(AMQSession.java
> > > > :462)
> > > >         - waiting to lock <0x88ff0240> (a java.lang.Object)
> > > >         at org.apache.qpid.client.AMQConnection.closeAllSessions(
> > > > AMQConnection.java :752)
> > > >         at org.apache.qpid.client.AMQConnection.close(
> > > > AMQConnection.java
> > > > :673)
> > > >         - locked <0x8b85a1a0> (a java.lang.Object)
> > > >         at org.apache.qpid.client.AMQConnection.close(
> > > > AMQConnection.java
> > > > :659)
> > > >         at org.apache.qpid.test.unit.client.forwardall.Service.close
> > > > (
> > > > Service.java:71)
> > > >         at
> > > > org.apache.qpid.test.unit.client.forwardall.ServiceCreator.closeSC(
> > > > ServiceCreator.java:57)
> > > >         at
> > > > org.apache.qpid.test.unit.client.forwardall.ServiceCreator.closeAll(
> > > > ServiceCreator.java:66)
> > > >         at
> > > > org.apache.qpid.test.unit.client.forwardall.CombinedTest.tearDown
> > > > (CombinedTest.java:44)
> > > >         at junit.framework.TestCase.runBare (TestCase.java:130)
> > > >         at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > >         at junit.framework.TestResult.runProtected(TestResult.java
> > > > :124)
> > > >         at junit.framework.TestResult.run(TestResult.java :109)
> > > >         at junit.framework.TestCase.run(TestCase.java:118)
> > > >         at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > >         at junit.framework.TestSuite.run(TestSuite.java:203)
> > > >         at junit.extensions.TestDecorator.basicRun (
> > > > TestDecorator.java:22)
> > > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message